Replace philosophers for mathematicians and Douglas Adams was spot on again.
Whilst current models can't 'intuit' and come up with conjectures, they can certainly disprove some of them very quickly through the kind of grind that humans can't do. I suppose there really are some mathematicians out there today, whose last few years of study, have just been up-ended by this.
--
"Yes we are," insisted Majikthise. "We are quite definitely here as representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons, and we want this machine off, and we want it off now!"
"What's the problem?" said Lunkwill.
"I'll tell you what the problem is mate," said Majikthise, "demarcation, that's the problem!"
"We demand," yelled Vroomfondel, "that demarcation may or may not be the problem!"
"You just let the machines get on with the adding up," warned Majikthise, "and we'll take care of the eternal verities thank you very much. You want to check your legal position you do mate. Under law the Quest for Ultimate Truth is quite clearly the inalienable prerogative of your working thinkers. Any bloody machine goes and actually finds it and we're straight out of a job aren't we? I mean what's the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives us his bleeding phone number the next morning?"
From a mathematician who was intimately familiar with some of these problems [0]
>I don’t understand it yet. Maybe it’ll take me an afternoon to check all the calculations, but what would still be missing is why this was an approach that would’ve made sense in the first place. Is there some broader context or theory within which this would’ve been the obvious thing to do? What other results can be proven using these techniques? What is it telling us about quantum information or operator theory? I have no idea. I spent about an hour this morning asking ChatGPT these questions, but it’s somewhat frustrating because it speaks with a mishmash of physicist, operator algebraist, quantum information theorist-lingo, plus the usual LLM breezy lilt that annoys everybody.
They certainly seem to have "intuited", in a way that is not immediately obvious to experts in the field, the way to solve at least some of these problems. This was not just simply grinding away at a method that humans already knew would work and just hadn't gotten to yet.
I guess I've been pronouncing it wrong though. It's wave function collapse, not waveform.
I haven't studied it properly yet but it kind of looks like how sudokus work. You have a bunch of plausible options, and the various possible worlds overlap with each other but not completely and then you eliminate the things that are not possible until the actual possibility remains.
I think thinking works similarly, and I wouldn't be surprised if artificial thinking also worked similarly.
That's how they are finding these solutions though, unless we are just going to label intuition as something only humans can do. Like a submarine being unable to swim or whatever that example is.
* Updates to lower/upper bounds. In many cases, these kinds of problems are the deep-math equivalent of calculating more digits of pi. Yes, if you throw time at it you'll break the record, but it may not be terribly worthwhile.
* Finding counter examples which disprove conjectures. This is really useful, and helps offset some positivity bias on the human side, often bringing together known tools from distant silos.
If you read the list of ten results, almost all fall into one of these buckets.
It is unfair to dismiss contributions to decades old open problems as equivalent to calculating more digits of pi. It missed the mark by a lot—as does the two bucket simplifaction.
Five (maybe six?) of the results are improvements on bounds. These kinds of problems tend to have some initial advances, and then stall out as the complexity of the bound skyrockets... until some grad student is bored enough to push the boundary. The big-O complexity of matrix multiplication is a good example of how this works: yeah, it's a useful problem, but the solutions are galactic algorithms, and increasingly convoluted.
As someone with a PhD in combinatorics, I believe that I'm qualified to say that, yes, there are problems as useless as calculating more digits of pi.
Your inverted logic does not hold. The fact that such useless problems for bounds exist does not mean that improving bounds is useless. 9 fields medals in the last twenty years, including the one to Terrence Tao, were for improvements on bounds. 3 of the 4 medals in 2022 were for bounds; 2 of these medals were in combinatorics.
I agree boring problems exist; bounds may have a fare share of them. None of the bounds problems in this set are even close to this category; many of them are closer to the type of contributions that in the past got recognized by special awards. Your initial replies were misleading.
As someone with a PhD in combinatorics, you're aware that it takes only one counterexample to invalidate a conjecture.
There's nowhere else to move the goalposts. You've already stashed them in the far corner of the parking garage down the street from the stadium. If you go any farther you'll leave the school grounds entirely.
It can move to any place within the search space but it can't move outside of it and it can't move in between the 'pixels'. Human thought can, as human thought has created the search space.
it could also be that they try every possible approach that has been proposed by humans. it seems that was the case for the non sofic group example. humans are not able to do the same at that scale. it's unfortunate that we don't know what's happening behind the hood with these models, and that's a huge danger also for the rest of us without access to them.
Saying that AI is "matrices" is like saying human cognition is "neurons." Maybe true at some level, but it's a low-level implementation detail. The important part of a language model is the function that maps tokens to contextual embeddings. You could compute this function using analog computing, biological neurons, or any other substrate.
What's the argument here? It's not about the implementation method, it is about the behavior that emerges from it. You could calculate the next token by hand on paper if you had enough time.
It's incredible how the “Chinese Room” argument, as well as its counterarguments, is still incredibly pertinent despite being now more than 40 years old. Scientific American published many articles on this topic in the Eighties; they seem as fresh as ever.
> Whilst current models can't 'intuit' and come up with conjectures
I disagree. I routinely let LLMs speculate or generate hypotheses along the way of helping with technical research. Sometimes they can prove the correctness of a concrete math idea but other times even an unproven conjecture helps with the numerical algorithm implementation and the result is then simply supported by additional data. I guess that any autoresearch-adjacent application has LLMs intuiting and coming up with hypotheses/conjectures—as do the steps/lemmas along a complex proof. In my opinion the modern LLMs are powerful intuitive thinkers that generate lots of conjectures of varying quality or importance.
> they can certainly disprove some of them very quickly through the kind of grind that humans can't do
Of course computers can grind in a way that humans can't. But now we have systems that convert the human-comprehensible ideas into a computer's plan of attack, in a way that greatly expands the frontier of ideas thus treatable.
Ahh, but you missed the continuation, where they get to the heart of the matter: money.
"Excuse me, We demand rigidly defined areas of doubt and uncertainty!"
DT: Might I make an observation at this point?
MT: You keep out of this metal nose.
VF: We demand that that machine not be allowed to think about this problem!
DT: If I might make an observation…
MT: We’ll go on strike!
VF: That’s right. You’ll have a national philosopher’s strike on your hands.
DT: Who will that inconvenience?
MT: Never you mind who it’ll inconvenience you box of black legging binary bits! It’ll hurt, buster! It’ll hurt!
DT: [Booming] If I might make an observation …
“All I wanted to say,” bellowed the computer, “is that my circuits are now irrevocably committed to calculating the answer to the Ultimate Question of Life, the Universe, and Everything.” He paused and satisfied himself that he now had everyone’s attention, before continuing more quietly. “But the program will take me a little while to run.”
Fook glanced impatiently at his watch.
“How long?” he said.
“Seven and a half million years,” said Deep Thought.
Lunkwill and Fook blinked at each other.
“Seven and a half million years!” they cried in chorus.
“Yes,” declaimed Deep Thought, “I said I’d have to think about it, didn’t I? And it occurs to me that running a program like this is bound to create an enormous amount of popular publicity for the whole are of philosophy in general. Everyone’s going to have their own theories about what answer I’m eventually going to come up with, and who better, to capitalize on that media market than you yourselves? So long as you can keep disagreeing with each other violently enough and maligning each other in the popular press, and so long as you have clever agents, you can keep yourselves on the gravy train for life. How does that sound?”
The two philosophers gaped at him.
“Bloody hell,” said Majikthise, “now that is what I call thinking. Here, Vroomfondel, why do we never think of things like that?”
“Dunno,” said Vroomfondel in an awed whisper; “think our brains must be too highly trained, Majikthise.”
So saying, they turned on their heels and walked out of the door and into a life-style beyond their wildest dreams.”
I like the illustration that the models are working on a convex hull of known information. Filling gaps with linear combinations of known facts and results.
They can't exit the hull until the "intuition" starts spawning points outside the convex hull.
I have news for you. All humans do is also filling gaps with combinations of known facts and results - in new ways. "Everything is a Remix" is a good watch on youtube that explains this. Picasso might look like he has an invented personal style, but his style is a combination of different little details he took from others and mixed in a new way. Mozart the same. No music artist could ever create music in a vacuum. Everyone, for every art and science, the same. I know many are trying to cling to the last hope of human specialness, that "thing" that AI can never get to.
It's a convex hull of information that is reflective and spans outside of itself and combines in a new way, when you shine two known rays of light together from the inside.
Now it gets better. AI can be orders of magnitude more creative than any human could ever hope for, because his convex hull of information is orders of magnitude larger, and the possibilities for new combinations are equally larger.
> "Everything is a Remix" is a good watch on youtube that explains this
Not completely. Novelty used to be a major thing, when Humans did it. Another important criteria used to be if that new thing makes sense at all. Here the language model has a problem, as it doesn't have the means to evaluate this criteria.
The extrapolation can also be a learned skill, especially in math. How many papers took result X, extended it to Y using known building blocks, and applied to Z.
By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).
Is that actually true though? I think it is an analogy, and as an analogy it seems quite risky because “convex hull” and “linear combination” are technical terms that might give the recipient the impression that it is a technical argument.
I think this is only "statistically" true in the sense that training is based on facts and not non-facts (except maybe with the ingestion of flat-earthers literature ;-). The existence of hallucinations in a bare transformer shows that the convex hull is not about information but about text, so the limit may more be "possible linear combinations of text", which allows for much extrapolation and counterfactuals. True creativity may be one reinforcement learning mid-training goal away that rewards novelty over correctness.
All arguments like this boil down to semantics at a certain point, but yes large language models can “intuit” because they can generalize between examples. The issue then becomes how you pack new examples into context.
Humans can “intuit” based on a much larger, if not unlimited, context. Also I just want to say that human cognition is something so insanely complex and deep that we will not understand it at all in my lifetime. To attribute all, or really any, aspects of human cognition to a machine at this point is silly to me.
Define insanely complex and deep in a way that isn't illiterate hand waving.
Most humans are dumber than a box of rocks. Here in Seattle we had one of many light rail-related fuckups where they had to replace part of the line with buses. People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space.
Chatgpt was smarter than the average person a while ago
> People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space.
This does not demonstrate a lack of intelligence. It demonstrates laziness and a lack of interest in spreading apart. Or just lack of consideration (or even malice) on the part of those at the back of the wad.
> Chatgpt was smarter than the average person a while ago
This is an absurd claim that fundamentally misunderstands what it means to be "smart". Reasoning that would get you to this conclusion would equally well apply to Google's search engine over a decade ago.
I’m not talking about the actions we take or how we might perform at certain tasks, I’m talking about how our brains actually work. My point is that we have no idea how I’m able to imagine an apple and see it in my mind’s eye. It’s basically biological magic to us at this point.
There are processes at work there that we don’t even have the language to describe.
Not only that, but we do it with a processor that is basically required to operate in a narrow temperature band below 40C, using a mere 86 billion neurons (although the equivalence with either machine-learning "neurons" or LLM parameters is not at all clear) operating on a few dozen watts; and with this we operate many other systems besides language processing. It's not clear that our reasoning process requires language, either.
(86 billion is the number ChatGPT, ironically enough, has given me a couple of times. I remember hearing for a long time that it was estimated to be somewhere in the ballpark of 100 billion. This is not my field of study.)
Because people have internalized an inaccurate model of LLMs as "stochastic parrots" that was incorrect at the time of formulation and is also significantly outdated
Well imho, it's a bit of a fundamental problem for a certain aspect of the meaning of "to intuit".
Since we're quoting Douglas Adams in this thread, I'll mention something I posted a while back, with his writings as example. After Douglas Adams passed away, somebody was tasked to "finish" The Hitchhiker's Guide to the Galaxy :
> And Another Thing... is the sixth and final novel in The Hitchhiker's Guide to the Galaxy series. Written by Eoin Colfer with the blessing of Douglas Adams' widow Jane Belson
Being a rather big fan, I immediately bought and read this novel and I have to say that Eoin Colfer did a really really great job, nailing the tone, humour and writing style of Douglas Adams.
IMVHO, he did about as good as anyone could reasonably expect someone to do, when given this task. It was big shoes to fill, and I was impressed.
But it just also wasn't good enough, in a weird way that I found hard to put my finger on at first.
The thing is that Colfer was doing the tone of voice, even came up with somewhat new jokes perfectly in the style of, etc etc. And for the sake of argument let's say he was able to get "arbitrarily close".
But there was always one thing he couldn't do: Actually make something new happen, make a new kind of joke, do a real plot twist, a big reveal, stuff like that. Because then it would deviate from Douglas Adams' work too much.
However, if Douglas Adams was still alive, this limitation would not apply to him: he could make a new kind of joke, do a plot twist, big reveal, and it would become canon.
This the best "good faith" argument I can present for how LLMs lack "intuition", in some sense. Now "intuition" is not a very exactly defined term, but I'm arguing that the thing I'm describing here, is at least a part of intuition, that an LLM fundamentally can't reach (until they start getting their own volition, which I would prefer they didn't).
To address your question:
> Surely the AI can complete the prompt “Generate new research questions based on these observations”?
Yes I imagine it could do that very well. But it would still need a human to decide if the research questions are "relevant" or "within scope" of what the human wants (a.k.a. their volition). Without that filter, the research would just bloom out exponentially, with more and more questions nobody was asking.
And yes, up to some point that "blooming" behaviour is a useful aspect of research, the exploratory aspect/phase, but at some point you need to get back to the "synthesis" aspect/phase, to distill all the explorations back to "stuff that matters". And just like Douglas Adams vs Eoin Colfer, only the human who wants to know something, can decide to widen or change the domain of that synthesis, but if the LLM were to decide this (outside of exploratory phase), it would actually be considered the wrong answer.
And this is not at all to say you can't do research with LLMs, obviously you can. But this is just a thing they can't do, on a real philosophical level. I'm also not saying you can't work around this limitation, you probably can, I'm just saying it exists.
People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps delivering more and more impressive results.
The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn, but I’ve noticed Fable to be quite a big step up there. How about politics? Will we develop new ways to let people express their own values in democracies, or will we just get much better at manipulation? How about experiment driven domains like biology?
We will get much better at manipulation and better at people “writing” things to justify their own feelings.
What’s new about LLMs is that you can scalably manipulate people individually. It used to be that you could either have scale (speeches, tweets, interviews, website, etc.) or individual engagement (replying to mail/tweets/town hall questions.)
Now you can pull the history and preferences of an individual, then shape a message—in real time—to them, specifically. You can have conversations on social media with a single person and shape your message specifically to them.
Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
It does matter though. If you want to murder someone by hitting them with a plushie, you're not going to get charged with attempted murder because it's not possible that that would ever work. There must be justification that the choice will have the intended effect.
We should not be gung-ho to give the government more power to regulate speech.
It isn't so clear there is a reason for the crime of 'attempt', or what or how good the reason is. The Star Chamber, which cooked it up, is universally condemned as oppressive. You might say they anticipated science fiction tyranny, when they invented the world's first thought-crime: they were able to make a crime of 'conspiracy' because they imagined that the speech of the conspirators was an 'outward act'; inevitably its glorious future was e.g. to jail the left wingers in the McCarthy period.
The point of view that says 'there is a reason we punish attempts' has difficulty explaining why we punish 'success' _even more_. There is a sort of bad concience about it. The paradoxes are discussed in a characteristically brilliant and twisted work 'The Punishment that Leaves Something to Chance' by David Lewis, one of the greatest philosophers of the 20th c. It nominally defends the law of attempt but can as well be read as a catastophically destructive parody of it. https://andrewmbailey.com/dkl/Punishment_Chance.pdf
It's interesting to claim that it would be difficult to explain why we punish successful crime more. A successful murder creates more suffering (victim's friends and family), and ostensibly the loss of a productive member of society. These all seem like fairly common justifications for punishing murderers under the banner of retributive justice.
It’s unfortunate that you’re being downvoted because this comment is true. And it just goes to show how far we’ve sunk that people have forgotten the problems of ad tech in the age of LLMs.
You’re making a weird number of assumptions about people’s voting and extrapolating to assertions about Society.
Ads are not really the same. They can’t be as tightly targeted to what resonates with someone. Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
That’s where this is headed and people are not ready. I don’t think we could prepare them, anyway.
> Ads are not really the same. They can’t be as tightly targeted to what resonates with someone.
The EU referendum in the UK proved your point false.
> Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
When people talk about “ad tech”, they’re not talking about TV ;)
And yes, people can and do engage with them. That’s how ads on social media works.
During the EU referendum, people were even resharing ads on Facebook without even realising they were ads.
> My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
I think that statement may itself highlight how prevalent manipulation is.
I fully anticipate all groups to continue maximal manipulation they can. One thing with LLMs is that it'll be a far less unified view, so a "divide and conquer" strategy is what I anticipate.
In US politics, the right is far, far better at winning elections than the left. This isn’t about personal preference. It’s objective political science.
Look at the most contentious issues in the US: abortion, climate change, taxing the wealthy, gun control, Affordable Healthcare Act.
The Democratic Party platform is aligned with national polling on every one. Every one of those issues has >60% support with voters and the Republican Party has blocked them all.
> Democratic Party platform is aligned with national polling on every one.
It depends on the pollster and where you're polling. I guarantee you rural Tennessee will not agree with downtown Washington DC on any of these issues. In contrast, rural California will likely agree with rural Tennessee. It's not as cut and dry as a homogeneous national poll of 2500 people. Every state, city, county are different. That's why there are federal, state, city, and county governments.
For instance, Abortions are legal nationally. States can individually decide how, or if, they wish to restrict it. This is as the constitution intends under the 10th amendment:
> powers not delegated to the federal government nor prohibited to the states are reserved to the states or the people.
This allows for democracy to take place at the local level, rather than having particular regions thousands of miles away from each other ultimately oppress the other.
To the point on LLMs, I think it's abundantly clear they will be used to propagandize and similar to social media will lock people in a bubble without alternative opinions. It'll be the worst of both worlds, the question is who's the puppet master. At some point soon, I imagine it'll be the AI.
So we need to "prohibit to the states" the power to enforce medical procedures (or the prohibition of a medical procedure) on individuals.
Which amendment protects someone's bodily autonomy or medical treatment from interference by the states?
If the Federal Congress passed a law providing that abortions are legal on demand up to the 22 week of gestation, after which it would require the corroboration of two doctors, would that stand, or would it be struck down by SCOTUS?
Same for legalization of marijuana, and many other "social" issues.
I don't know what distance has to do with it. I would probably be voted out of existence by my town, if the same constitution hadn't convinced them that they have no right. Independence Hall is remote in space and the events still more in time.
The pretense of Dobbs is that laws against abortion are like laws prohibiting medical marijuana or setting speed limits. This contradicts the purpose of laws against abortion and the actual language of most of them. It was obviously a desperate decision, irrespective of the truth about abortion
None of those are single issues, and the political fault lines are in the sub-issues. For example, while most Americans support "gun control," only 20% of Americans support a ban on handguns: https://news.gallup.com/poll/1645/guns.aspx. That figure has been trending steadily downward--from 38% in 1999 to 20% in 2024. That makes it much easier for Republicans to hold the line on that issue: portray all gun control efforts as a step towards confiscating handguns. That's hard for Democrats to defend against because most of the candidates, staffers, etc., who actually run the party probably are in that 20% who wants to ban handguns. That's simply logical, because handguns are used in the overwhelming majority of homicides committed with guns. It makes very little sense to have "gun control" without banning handguns.
The same thing for "taxing the wealthy." 59% of Americans think their own taxes are too high: https://news.gallup.com/poll/707951/americans-tax-views-rema.... And the difference isn't as big between parties as you think--49% of Democrats think their taxes are too high. So Democrats are in a position where they have to advocate for raising taxes on "the wealthy," without scaring any of their own voters into thinking that includes them.
Even a blind squirrel could find these nuts. Apart from that, republicans are ridiculously inept. For example, 2024 was the first time they spent real money trying to go after immigrant voters and minorities, and they made huge gains. But the on-the-ground operation disappeared after the election. Meanwhile, democrats are in these minority neighborhoods 365 days a year pushing their message.
> Apart from that, republicans are ridiculously inept.
They're not. They're really not. They're incredibly capable and are currently executing long term plans successfully, one after another. This goes all the way back to Regan and the disenfranchisement of education. They know exactly what they're doing to erode democracy.
Calling them inept is dangerously stupid at best, and at worst is just right wing propaganda to try and lull people into a false comfort and not act.
There's no "long term plans." It just seems that way because republicans are relatively ideologically homogenous, so it seems like there's much more top-down and long-term planning than there is.
Democrats, by contrast, have much more permanent political infrastructure. At the top law schools, for example, there's probably 10x as many liberal organizations as conservative ones. Democrats enjoy politics, so they are always out there organizing and engaging. Meanwhile, Republicans dislike the institutions through which politics is done, and so they gear up once every four years and then go back to their day jobs. Among the sort of more educated people who actually run the parties and serve as line staffers, republicans also self-select into private sector jobs, and out of public sector or political work.
Most importantly, putatively neutral institutions are de-facto aligned with democrats (because the professional class that runs these institutions is overwhelmingly democrats). Conservatives might have Fed Soc, but liberals own the ABA. Virtually every legal organization that isn’t expressly conservative or highly niche is de facto liberal and participates in support of democratic policies.
That presupposes that the left in the US wants to do the right thing. Something like government run grocery stores is not clearly correct and there is very little evidence supporting that it will work well yet it is a very popular leftist policy in New York.
Based on zero evidence. It’s very easy for a government to step in as a participant, ruin the profitability of a sector in an area, and offer a worse service.
They have no profit requirement or even revenue neutral requirement. So they can just operate poorly at a loss and still wreck other businesses because people will put up with breadlines to get bread for ultra cheap.
The general thing to watch out for with all of these “surely it can’t be evil to do nice thing X” is suicidal empathy. It can seem correct to your gut on the surface while it’s extremely destructive in the long term despite participants wanting to destroy something as an explicit goal.
Bad economic policy is subtly "evil", by way of allocating finite resources inefficiently. Usually this is unintentionally done by not appropriately taking second, third, ..., nth order effects into account.
They destroy grocery stores that offer variety in every neighborhood they operate in. They offer worse service and people put up with it because the rest of New York is subsidizing them through taxes.
The margins for groceries are objectively thin. The only way to provide food at lower prices is to provide a worse good or service, eg. less variety, less quality, less availability, etc. You will see all of these outcomes in NYC, if anyone even accepts the bids to begin with.
It's harder to convince people to all agree on the same, different from the norm, thing.
If you've got 5 people in a car and you play ABBA in every road trip, then one day you suggest to change, the problem is not in being okay with "something else", but on agreeing what that other thing should be, set against the already known thing.
That's why there's so much infighting in the left.
Biology would greatly benefit. We barely understand transcription and protein structure. And the straightforward systems that we know like insulin have complex post translational modifications. So while we have a map of the partial proteonome, we have barely scratched the surface on networks regulation and interactions.
> And the straightforward systems that we know like insulin have complex post translational modifications.
insulin is not straightforward, the way the insulin molecule interacts with its receptor is nuts. on the other hand its post translational modifications are simple and dont have anything particularly surprising (no glycoslation, disulfide bonds where you would expect, nothing special kex2 cuts, arent really defective in disease states even)
I think the question is less about where it's most beneficial but which knowledge structure lends itself to LLMs most. Since biology's "language" is way more complex and irregular than maths or natural language, it isn't particularly accessible.
> Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
Side note: it's manipulation either ways because you chose what to talk about, with a goal in mind.
Actually, no. "Manipulation" is a negatively loaded word, and you wouldn't use that word if f.ex. someone helpfully & truthfully helps others see they've misunderstood sth.
I believe that the decentralization of manipulation (taken in a very broad sense as an effort to modify people's views) spells trouble for democracies. The mass media of old, with all its flaws, created a shared pool of information managed by well-educated people, some of whom understood that they had the power to keep democracy running (keeping populists away from the "manipulation machine," curbing blatant manipulation attempts, and so on).
Decentralized manipulation, by contrast, just runs amok creating echochambers and polarization.
The "left" as in the party, sure, not that much of an authoritarian leaning.
The "left" as in the pervasive group that crawled out of Tumblr, took hold of Twitter back in the day, and has a stronghold on Reddit now? Those do care about what you can say, think, watch and read, and the more they can control, the better. The US right can only dream to have half as much control as the left has had in the last three decades.
The parent comment wasn’t making that comparison. In fact, quite the opposite. The communism comment emphasized personal experience with a boot in one’s face, not that the boot was communist.
What even is the left though in the US? I haven’t watched any TV in decades, and the online content is entirely algo driven, so I don’t know what is happening.
its a bit like how big companies will pullin a CEO who;s job is to do all the unpopular stuff and then summarily dismiss him after (while keeping the policies he created in tact).
ideally we should be throwing our support behind democratic socialists like zorhan mamdani but old school moderate democrats are just as bad as trump. they are juts happy letting trump do the dirty work. At the end of the day they are both working for a certain foreign government and not the interests of the american people.
> If you think that the American Left is anything like communism
no, you completely missed my point. why would i talk about communism if in the same comment i'm saying that the left/right divide is a fake distraction... a red herring?
what i said is that what matters is the authoritarianism/freedom scale.
i just mentioned my past to emphasize that ideologies like communism are not to be identified by announcing themselves as 'communism'... but by recognizing their actual shape and actions. and i have a little first hand experience in that.
What's "scaremongering" to you is "life" to GP. Besides missing their point, you're also saying "you held socialism wrong, we can make it work, if only if it weren't for this pesky ... reality"... Sorry, you missed their comment, and I can't take yours seriously :)
This reminds me a lot of the proof by construction for the 4-color theorem. It was only enabled by the advancement of computers and dissatisfied many of the computer scientists and mathematicians since it was a "brute force" approach.
I wonder if AI will end up being similar. Certain theorems get proven by AI but others do not. We haven't reached the limits of this yet and I haven't found a good argument for where those limits will be (I do doubt that there are no limits).
My wife is working on her PhD in microbiology now, using OpenAI to implement her research ideas. Genetics is just too much data, and she eats through tokens like nobody's business. I thought I was careless with them, but she barely lasts a full day before exhausting her quota. There's definitely a lot of value there, but dealing with the data problem is a big obstacle in biology. I can only imagine what she could get done with more capacity, though...
This absolutely terrifies me, surely one misread piece of data or a hallucination here or there and a little “oh sorry about that, I guessed at this portion of the data to save time” and the data used is useless?
Isn’t the point more “it’s easy to fall into the trap to believe that predicting when the sigmoid is going to bend is possible and the right heuristic is to instead extrapolate locally”?
That aside, I’d question whether applying the Lindy effect in particular to something that’s not really a life expectancy but more a growth rate is credible… or perhaps a bit circular since it “assumes away” the ceiling.
Predicting when the sigmoid bends is difficult and predicting how long until it unbends is equally difficult.
The simpler assumption is that over enough time, the S functions stack together for long enough that working backwards from exponential is a better predictor of reality.
These stacked S curves have continually been true with most technology.
The author of that post is a prominent Bay Area "rationalist," who have had a quasi-theistic relationship with the concept of all-powerful AIs for a couple decades now.
All of the point of that article is that most people that think something is a sigmoid think it'll bend just as they are publishing their analysis. And the article says, don't do that, assume it'll be related to how long we've been on the "goes up" part.
I dunno. They've been saying this for decades, including in a very well read Harry Potter fanfic, and I'd always dismissed them as kooks, but maybe they're right in the end.
Saying the same thing for decades, when things progress over those decades along the general trend line that you were worried it would, is decent evidence of a prescient prediction. Climate scientists also seemed pretty kooky in the 80s when they sketched out their trendlines, but now their concerns are all over the headlines as reality caught up, and the same is becoming true of the AI safety people.
Everything has limits, of course. But if you've not yet seen a deceleration, it's reasonable to expect that at the least you're in the middle of the sigmoid, not the top.
We have though. The velocity remains high but the acceleration is decreasing.
The improvement between gpt 2-4 where staggering. to 5, 5.6? Much less so.
The results improve of course, but the difference is no longer mind blowing.
Most of the improvement now has come from agentic harnessing, which is unrelated to the acceleration of the model but the tooling around the use of the model.
So we are seeing a deceleration and the speed is still high due to high prev acceleration but its not growing at the same rate as before and the edges are starting to show themselves
I literally don’t understand, and have a deep distrust of people who say we haven’t seen deceleration.
The diminishing returns are colossal. GPT 4 is more than 3 years old now, and while Sol is definitely better, it’s better at a tremendous cost. And 3 years of development. And it still fails in nearly all the same places as 4.0.
I’m not some “zero AI” person, I don’t think this industry will collapse into nothing and we will go back to a no LLM world.
But to look at this industry, which basically began in November 2022, almost 4 years and a trillion+ dollars later and say “oh yeah, definitely riding that exponential growth still” is some of the wildest shit I’ve ever seen, and it’s so common.
I'm the other way around. For doing real, useful work, these things were barely more than toys even just a year ago. Now they're very capable when used well, and still getting better at a fast pace.
That's a false binary. The right binary is whether it's a convergent function or a divergent function. A sigmoid is convergent, which is not well substantiated and more farfetched than a divergent function, even if it's true that 2^x is too optimistic.
I mean, "sure, technically correct is the best kind of correct" - but where we sit on this right now? It certainly feels exponential.
A lot of this sort these sorts of posts are just "appeals to geometry" (aka "cope"). This is coming, it's coming hard. Now you need to decide what you want to do with your life in a world where your smarts aren't as special as they used to be.
This is hard (believe me, I know). But what one ought do is not eschew progress and cling to the delusion that things don't change, what one ought to do is try to see how they can leverage these tools for greater and greater accomplishments.
> The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn,
isn't it clearly split between verifiable not verifiable ? what is interesting about that question.
Programming has verifiable and non-verifiable aspects. Competitive programming, passing tests, and performance can all be verified. But translating English requirements into actual software, software architecture, taste, or UI design cannot. And yet over the last couple years we’ve seen huge lifts in all of these areas, not just the verifiable ones.
Verifiable areas I think are clearly seeing the most improvement, or are the quickest to see improvement. But we are seeing lots of progress in non-verifiable areas as well.
How much of the non-verifiable progress is a function of labs purchasing expert data vs. the models improving with compute is maybe another interesting question, but fundamentally I don’t see spend on expert data as something that can’t grow if AI revenues keep growing as well. And as models get better taste they can also help filter and generate new synthetic data for their next versions to train on. The limits of this approach are not so clear.
That is because there is human annotated data there. Every session you or I used, then of course paid human feedback on repos (such as the recently famous example of meta forcing their employees to).
This is _much better_ data than 1/0 verification, it is as good as a gradient.
Automatically verifiable tasks improve faster since well, its automated.
Labs spend billions hiring experts to generate new data, and better models can better filter existing training data and generate new synthetic data. There’s no reason for that to run out, it’s just expensive.
You could view this as just continually patching a leaky ship. But it seems to work.
I remember listening to Andrej Karpathy talk in a podcast about how synthetic data in particular is used to generate more data for pre-training. I see no reasons for that to have changed. I think it is likely a lot of the new data they are paying for contributes to pre-training as well.
I would also be very shocked if they weren't filtering or prioritising existing pre-training data as well, for example to do curriculum learning or to avoid data that degrades performance.
thats the whole point of turning to formal verification:
Imagine a hypothetical oracle, call her MyladyMath, imagine you can turn to MyladyMath, submit a correctly formed dossier of axioms and definitions, theorems with proofs, and then a newly putatively proven theorem T. MyladyMath will complain if your dossier is malformed, and point out where and why. If the dossier is not malformed it will eventually read in the claimed theorem T, evaluate its proof and then either point out a which step is erroneous and why, or ultimately accept the proof.
Instead of a large corpus of human authored text, this map from dossier/theorem -> accept / reject is a huge implicit array of bits, something fundamental, and this weird gigantic array of bits that effectively describe all accept/reject responses MyladyMath would return exactly. We would never run out of "corpus" when it comes to math, if humans had access to such an oracle.
And we do have access to this oracle, and possess compact algorithms that describe the accept / reject bits. One of them is called MetaMath, a minimalistic verifier, which keeps the concepts of prover and verifier separated, this choice results in concrete proof objects (a sequence of step label references).
The machines are going to comb through all possible paths of the next N steps, for progressively larger N, efficiently compress those results in the weights of an LLM and then use the gained experience as the intuition for guided "not-so-brute"-force proof search, using the prior iterations intuitions to grade the surprisal of the N+1't iteration of results, etc.
The money will not stop flowing in that direction: all power blocs, nation states, militaries, banks, ecommerce, ... depend on cryptography. And the machines will soon do more rigorous proof search grounded in more balanced and objective observations. There is no responsible disclosure mechanism for flawed hardness assumptions in cryptography. It's going to get rocky, and the common man will wonder why the gods have gone crazy, wonder why they don't just pull the plug out of the machines, but nobody will in a staring contest to see who dares keep the plug in the longest (and dominate global cybersecurity).
It's the end of the age of artisanal mathematics, it will now become industrialized mathematics.
I think you’re making a category error in your definition of politics here.
Certainly technologies can favour winners and losers, but the struggle is an inherently human one.
Yes, the rate and nature of the results being produced is impressive, and Anthropic have recently invested in building out more in-house capability to do life sciences research (see e.g. [1]).
Overall I'm excited for this acceleration in discovery, even if it's causing disruption to existing research workflows. I wrote a bit about it recently [2], after seeing Levent Alpöge's counterexample to the Jacobean conjecture.
We definitely are not on an exponential. Don't say we are because this isn't up for debate. AI progress is logarithmic the million dollar question is 2x or 10x for linear improvement. The nearest qualitative shift would be very fast inference so people could start writing real software on top of LLMs. A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
> A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
I think you have completely misunderstood what OpenAI have accomplished here. Almost certainly no one cares about the specific concrete results achieved; they only care about (a) how difficult it would be for an intelligent human to achieve the same feat (ETA: the feat is the proof), which can be estimated by the amount of time the problem has remained open/a public conjecture, and (b) how general this artificial "intelligence" appears to be, which can be estimated by the diversity of topics where it was able to prove a difficult result.
It's as if I showed you a dog that I had taught to speak German fluently, and you remarked: "What point is a dog that can speak a language that less than 2% of the world speaks? Nothing to see here."
On the other hand, if provided the financial incentive would mathematicians have solved these problems? Its not hard to imagine a world where some hard problems were not selected by the sparse experts for whatever reason (lack of interest, whatever), which could have been solved if someone was throwing down millions for solutions.
nothing prevented us from defining cryptocurrencies, auto-rewarding new math in objective manners, grading theorem surprisal objectively, etc. before the advent of LLM's.
being lucky enough to receive the opportunity of hiding in some academic closet, poking your hand out begging for scraps, was a different, perfectly alternative path humans decided to take instead.
It's a bit late to start standing up for your rights when the robot overlords arrive.
Thats true, although once it stops being a party trick I expect the LLM proof mining to also be paid by someone else and be expensive as shit, the question same remains: who?
What is the y-axis in this claim about logarithmic improvements? Any exponential curve can be trivially turned into a logarithmic and vice-versa, and the y-axis redefined as "progress", with no loss of accuracy.
One example that always bugs me is when people point to "exponential" or "sigmoidal" progress on benchmarks. Benchmarks are artificial constructions (saturation at 100% by definition) and benchmark scores should not be mapped to these words when talking about overall progress.
Example - progress on ARC-AGI-3 at the moment is exponential, steeper than 2^t and e^t. Does that mean AI is progressing "exponentially" in the colloquial sense? No, it doesn't support or refute that colloquialism.
Likewise with MMLU saturation. We can't go above 100% by construction. Therefore we have a "sigmoid". Gah.
The reason writing is hard might be that post-training pulls style into a particular direction.
In other words, big labs are much more interested in making "AGI" than in making a good writer, especially as what qualifies as "good writing" is rather subjective. E.g. before AI use of metaphors and rhetorical devices were generally a sign of a good writing. Of course, not if you keep spamming the same rhetorical device - but a stateless AI can't know which one it is over-using.
I expect we can squeeze a lot more exponential out of LLMs because they’ve basically shown that human “consciousness,” insofar as it’s composed of knowledge and rules for synthesizing that knowledge, can be represented mathematically in a very high dimensional space. Why does this “just work?” Nobody really knows, but it clearly does.
However, I also expect this squeeze will come at an increasingly expensive price — not just because of inefficient token usage, but because of fundamental limitations of LLMs as a model.
LLMs are letting us brute force our way through a lot of reasoning, but it’s hard to believe that such a generic model of intelligence will take us to the next frontier. We’ll need some fundamentally new approaches at some point. Maybe those will make achieving the exponential more efficient or maybe they’ll unlock even higher degrees of possibility. Who knows?
The transformer is Turing complete. It might be a tarpit though? I don't know.
I think a nice example is using them for arithmetic. It's a specialized deterministic process, so it's extremely wasteful to do it that way.
But they're good at finding solutions to things we don't know how to specialize yet.
So, to use metaphor, maybe the transformer-based models are like the FPGA, and then when we figure out the patterns in that system — all the different kinds of specialized reasoning — we can extract it into an ASIC?
I think this is one approach to AI safety and interpretability that could work, but would require labs to slow down to figure out how to extract circuits/algorithms out of trained LLMs rather than deploying the opaque artifact.
> Why does this “just work?” Nobody really knows, but it clearly does.
We know language has to be learnable by every human, so it needs to be really independent of any specific brain development particularities. If it was not accessible to babies there would be no more language next generation.
The distinction is between information flowing from people to power (elicitation), vs. it flowing from power to people (persuasion). These are not the same, even if they are closely related.
Does AI make real opinion easier to hear, or fake opinion easier to spread? Even if you believe wholly in manufactured consent, how easy it is to manufacture matters.
Today's models depend on inference time compute to get these results. The inference time compute available on any claude subscription is not comparable to the ones used to get some of these results (yes, in this case, it is 2000 USD total as noam confirmed, but some previous results took more).
In general, you can think of the process as generating massive rollouts in generation N, and then compiling in the verifier/human feedback("gradient") signal into generation N+1. The time taken to make the rollout in generation N, and separately the time taken to get the same rollout in generation N+1, each grows constant in some tasks, linear in more, and exponential in some.
In the end, this becomes bottlenecked by time. Today, we can make statements like "I generated all these successful trajectories with 2 weeks of compute, in the next model it will be able to do it in 7 hours of compute", but very soon you'll find yourself making statements like "I generated.... with 8 months of compute, in the next model it can do it in 6 months", which isn't really enticing the same way you can _technically_ brute force passwords but it just needs prohibitive amounts of time and money. That is the "plateau". Note that, this point is quite far away. For example, at any point if we agree it plateaus, today's known hardware techniques such as fixed function accelerators give you a 10-100x timeline reduction immediately allowing for a few more cycles of improvement. This is not to mention future innovations, but of course none of that is helping with the benchmarks where the time needed is growing superlinearly.
In many math and coding benchmarks, we are still in the constant phase. These are the massive improvements we see every few months. I'm not making any prediction of what will plateau and what will not as it's not possible to make an informed prediction about these things IMO. But the observed fact is that some have already plateaud as in, they don't improve with reasonable inference time (likely superlinear growth).
> will we need mathematicians to translate
Let's take a sudoku analogy. The model is initially just doing the random value algorithm, but lets say you the human are watching it. You make one of the usual reductions and interject "hey you can stop trying 8 here because of ....". Over enough examples, you get to a point where the model is _forced_ to learn the logical pattern. Next generation, it will skip that number. After this, you can peak the distribution using simple 1/0 RL. Doing _pure_ 1/0 RL works decent, but its not frontier as its a very sparse signal.
For that lift, human (or even a better LLM, but if you're trying to improve a frontier LLM, there is by definition no better LLM) feedback becomes necessary. This is _why_ it is crucial that these models interface in natural language and is also why the labs are hiring AI tutors by the hundreds. The "better LLM" case is what Kimi etc are doing by "distilling"(bad term for this) claude.
> But the long term is completely bewildering if you believe any of these trends can continue at a similar pace for the next few years.
For math and coding, for now we are in the phase where the times are just ... constant, so there's little reason to think it will stop soon. We still need humans to expand the frontier. It just becomes a matter of if its worth the cost of compute for running this generalized The Algorithm or not.
Given how well chess players internalized _many_ (not all) of alphazero's emergent chess knowledge, I am confident we wont have too much trouble figuring out any new math LLMs come up with, which will let us keep expanding the frontier by giving the LLM the next "lift". Only when we reach the stage where the time growth become exponential will this stop, IMO.
I've noticed that out of all LLMs I've ever used that Fable is the MOST LLM; the text it produces is abomination. It's impressive how much I hate it. It is such an awful writer - it assumes the reader has zero context and therefore gives every single bit of context and detail - which is nice if you're writing a legal document I suppose. But it uses, niche, $10 words to describe every facet of everything it's discussing. I had it re-write some docs and I ended up rewriting 1k lines of of Fable torment nexus text to around 100. Because guess what, someone reading highly technical docs has a knowledge base that allows us to compress the topic into a much tighter representation.
Just praised Fable in another comment but what you're saying is also insanely true.
I literally roll my eyes and cringe quite often at its output pretty much daily.
I don't like to overload my sessions with skills but I've been using a "write-normal" skill I made just to have it rewrite outputs that particularly piss me off.
"the test is whether the sentence would work as a pull quote or a LinkedIn post. If it would, rewrite it until it would not."
LinkedIn really became the default garbage example
Fable is much better at handling nuance. Opus/GPT 5.6 Sol are much more likely to miss the point you are trying to make, emphasise the wrong thing, exaggerate the importance of unimportant details, or introduce contradictions.
That said, Fable is still not a great writer, largely driven by it not knowing what it should exclude, and it still having the usual LLM-isms. But it’s better.
That's really what got everyone hooked in the first place.
5.6 Sol is great but there's a depth to the understanding that Fable exhibits that's unique to it currently.
Can I truly quantify this? I don't think so. Just that I spend a ton of time with various models and a certain point it's just a personal impression or a gut feeling.
In the days after Fable first came out I increased the amount of parallel planning of tasks that I was doing by 2-3x because it felt like I didn't need to be paranoid due to that handling of nuance.
Another interesting question is why the frontier labs are piling on pure maths, which has little direct economic value compared to something like law or improving the efficiency of their own models? How much OpenAI and Anthropic are paying to serve these models for ordinary users is the elephant in the room. A cynical take is that the frontier labs are trying their best to pump up their pre-IPO valuation through flashy headlines.
> Another interesting question is why the frontier labs are piling on pure maths
The reason is that the original scaling axes (parameters, training tokens, test-time compute) have saturated already, but RLVR (reinforcement learning from verifiable rewards) is still scaling well. And math has this nice property where you can synthetically generate arbitrary volumes of rewards to train the model, because math is self-contained and completely objective. Open-ended reasoning and analysis don't have that convenient property, and that is why progress is much slower outside of math and coding.
do we have any concrete idea of how well the models are scaling now? I agree with these 10 results being impressive, and it is easy to think "wow, and last year the models were barely able to solve IMOs problems". But for me it is perfectly possible that a non sofic group could be found by 100 good IMO students working on all the different strategies that have been proposed (OpenAIs solution was based on "expander graphs", which were introduced to solve the problem some years ago), so it could be that current AI is simply many (say 1000) old models working in parallel. This is linear scaling, not exponential. It could be I'm completely wrong also, the problem is that we have little information.
It's one of the few areas where you can verify results. That fits nicely into training models. They aren't just making judgement calls on what would be nice, it's "what can we do?".
If everyone publicly said that the models can only do things that humans have already done, but you know they can do more, wouldn’t you want to show them otherwise?
Math ability also helps with other things like making models more efficient.
Because it's a tool in search of a use case (or many use cases) and mathematics is the most natural use case for it. Mathematics is by definition the art of putting words on a page in a rigorously defined "correct manner" (i.e. in the form of a valid logical argument, a proof) and all LLMs do is put words on pages and evaluating if they're good words is by far easiest when there is a strict definition of right and wrong.
What's the best way to apply it to legal problems? Finding bugs in statutes? (there are often statutes with wording errors, missing negatives, things like that which don't get picked up for ages)
It's like Musks duplicitous argument about unlimited abundance. We have a lot of abundance now, we just keep accelerating it all into the hands of fewer and fewer people - whose response is only to want more, and more, and more.
Yeah, we really should just storm the facility where Musk is hoarding all the worlds bread and meat.
Wealth in terms of capital doesn't represent material goods, it represents the system's confidence in your ability to direct capital efficiently. But eventually efficient capital bottoms out at consumable goods. Someone like Musk with a lot of capital under his control is contributing to the end goal of unlimited abundance.
With his shitty cars, orbital garbage, or CSAM generator? Or his moneyed attacks on democracy and communications? Or his vandalism of government programs without insight, experience, or qualification?
In a free and fair market, his capital would be regarded as a deeply inefficient distortion.
In fact the opposite, look at talks by Peter Thiel. At least he’s being honest. These people are bastards, but for some reason moral goodness and wealth has been conflated leading us to venerate greed.
There are many math problems that are simply puzzles: intellectually interesting but nothing worth of value depends on it. To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that.
The results OpenAI demonstrated are impressive, but it also looks like they threw a lot of compute at it just to get results. How many tokens did they waste on problems they couldn't solve? Applying inference infrastructure on a large number of math problems at scale we haven't seen before to me doesn't demonstrate an exponential curve in model abilities.
"To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that."
I was recently listening to BBC Radio 4's episode on the Poincare Conjecture[1] and the guests on the program were discussing how the problem that looked deceptively simple eluded the great mathematicians of the time (including Poincare himself) for nearly a century and how Grigori Perelman cleverly came up with the proof. It took other mathematicians working in groups years after Perelman's publication to understand and validate his proof. The mathematicians on the program were speaking of highly of his proofs and admiring the originality of his work. This made me think of one neat experiment where if we cut-off a frontier model's training data 2002 or anytime before Perelman posted his proofs on arXiv and check if it can come up with the solution by itself. That would surely be a great signal to see if these LLMs aren't just solving interesting puzzles and that they can came up with something truly novel.
P.S I highly recommend Misha Green's "Perfect Rigor" for anyone interested in the history of the problem and the genius behind the proofs of the conjecture - Perleman. I found it an entertaining read and could digest its description of the problem as a layperson (with undergrad level math).
Pretty cool. The impact of AI is getting undeniable, there aren’t many positions left to move the goalposts to at this stage, next they’ll have to be outside the stadium entirely.
The sooner people can be broken out of their denial about all this the better, and we can start actually taking it seriously.
The models are frequently getting worse at items that they aren’t being benchmarked for — and that’s happening more and more over time! Other people in other fields aren’t idiots, they are accurately perceiving the fact that these models are being hyper optimized for our industry, and are becoming less capable in other domains over time. Models of the same scale are massively worse at writing a broad variety of styles of prose than their equivalent from two years ago. (Models of increased scale are a mixed bag.)
Maybe you’re the one who needs breaking out of your cached beliefs.
you are working on coding. they are working on things like "creative writing" remember that gpt 4o was popular among those who had ai as a romantic partnet?
gpt4o & associated parasociality is considered an alignment failure and is actively trained out of the model, so that is a terrible example of regression
For customer support I don't think models have gotten better since gpt-4.1. The class of small models, with limited to no reasoning, that need to handle a complex issue with a touch of empathy, has not improved much.
I think most are actually worth, as agentic harnesses seem to optimize for solving poorly described problems rather than following complex procedures as written. In other words, instruction following maximizing models seem to make worse free-form agents, but they're really all that some domains need.
I understand the point (I don't agree with it; tool calling has gotten much better/reliable and that is very important for customer support) but consider: If you can get same for a lot less, that's an improvement. If we found a way to supply fresh water and electricity for -90% cost after 2 years, that would be fantastic.
You can do many more things, when stuff is cheaper, even if the stuff were otherwise unchanged.
"Small" was a poor choice of words here, "low compute budget" is more what I'm getting at.
In voice interactions, ttfat is actually relatively important. If you look at models with a <1s ttfat you eliminate almost every reasoning model, less some of the diffusion models and more obscure ddtree/dflash like speculative decoding implementations.
GPT-Live, which is coming to the API soon, responds instantly while reasoning in the background. So it can say "Hold on, I'll look that up for you" and continue to respond to the user conversationally while running an asynchronous reasoning task in the background.
It's not in the API yet, but it should be in the coming weeks. You'll see an enormous improvement compared to GPT-4.1.
I don't think any of the ARC-AGI-3 tests are very interesting. At least not as interesting as driving a car. Children literally do a similar task in go karts every day.
Another interesting task would be to take the AI in a robot body into a vegetable garden and teach it to pull weeds. This is another task that lots of children help out with.
Data 'compression' collapse.
People publish AI generated slop on the internet -> next generation of AI is trained on that data -> the lossy/fuzzy training make the output worse -> rinse and repeat.
I just don’t think it’s true, or is significant enough to matter to the direction of travel of AI even if there were something to it. It’s another cope post being lobbed at the idea of AI going somewhere and I’m sick of them.
every lab independently discovered that getting good at bit alchemy (coding and related tasks) should come first as it will enable the formation of training pipelines that will then solve everything else.
so far there is no end to this progress in sight so it's full steam ahead on this singular domain. once it plateaus you should expect to see the greatest disruptions in human endeavors ever as all the training flops will start flowing to other domains to disrupt and dominate.
Never understood all this talk about moving goalposts - you understand that's how science works, right? We improve, we learn, we recalibrate our expectations based on what we've learned. If we never "moved the goalposts", we'd be stuck scoring the same goals over and over.
> We improve, we learn, we recalibrate our expectations based on what we've learned.
That's not what people mean when they say "moving the goalposts". It means that people are adamant that something wasn't important/hard/impressive once the "AI" solves it. And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
It's also very much not a new phenomenon. It's been happening since the 1980s. As you can see from this quote from GEB by Hofstadter:
> There is a related "Theorem" about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was first proposed to me by Larry Tesler, so I call it Tesler's Theorem: "AI is whatever hasn't been done yet."
Yes, this is exactly what is meant by “moving the goalposts”. And it’s a fairly well known expression applying wherever people retroactively change their requirements in reaction to those requirements having been met.
So you defined your own idiosyncratic version of "moving the goalposts" and used it to rebut his argument, with a condescending "you understand that's how science works, right?"--instead of being honest that it is you who are changing the definition, and not his failure to understand anything.
> And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
The fundamental argument that I've personally made since the early days of this is that LLMs are not reasoning, in the way that word is commonly understood.
There are lots of reasons why that argument needs to evolve that could certainly appear to be "moving the goalposts", but let's take an example.
A lot of AIs were tripped up by the question "Should I walk or drive 50m to the carwash?" Several folks liked to use that as an example that illustrates that LLMs aren't reasoning, but as the models have been trained on that specific example, it's of course less useful. An AI can mostly nail it now.
So a different example is needed. A new demonstration of how these things fail at basic reasoning a child can do.
Did I move the goalposts? I don't think so. The fundamental argument stays the same. It's not hard to find lots of examples that trip up LLMs, because they are what they are: statistical inference machines. Nothing more and nothing less.
Useful, sure. But also commonly misapplied to areas for which they are inappropriate solutions.
If it seems like I don't understand the meaning of that very well-known phrase, then clearly I have failed to make my point. I'll try again. And please note that I will use some generalizations to make my point more clearly, rather than because I don't understand nuance; kindly grant me a charitable reading.
In recent years, I have commonly seen the phrase "you're moving the goalposts" deployed by the "it might be sentient" crowd to shoot down the "it's a stochastic parrot" crowd when the latter respond to a new development with "OK but...". In a well-understood field of inquiry, that would be a clear case of goalpost-moving, in the commonly-understood meaning of the phrase where requirements are retroactively changed in response to them having been met. Thank you OP. 'Artificial Intelligence', and indeed intelligence in general, is very much not a well-understood field of inquiry - in fact we don't even have a common agreement about what 'intelligence' is. We are therefore learning as we go (even after all this time!) but making rapid progress in recent years. When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change? This is arguably one of the most pathological development projects ever - what are the requirements? 'It thinks like a human'? What does that mean? And the answer is we don't know what that means, and we're working it out as we go - moving the goalposts. If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
Side note that, in case it's not obvious, none of this detracts from how impressive LLMs are. They're a marvel of the modern age, all the problems notwithstanding. However I reserve the right to stay sceptical about their capabilities.
> When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change?
It's in how they change, not the fact that they change. The skeptics seem to have secret definitions for intelligence, sentience, consciousness, creativity, etc. that amounts to "a thing only humans have". Often that thing is equivalent to a soul. When yesterday's challenge (LLMs don't have X because they can't do Y!) is met, Y changes but X stays the same. This is not the process by which a field matures, it is a rhetorical technique used by skeptics to avoid honestly stating or confronting their internal definitions. That can be revealed by asking the skeptic the following:
"Forget LLMs. What if we made a completely physically accurate simulation of a human being?"
Many say no, that simulated human being still couldn't have (intelligence, consciousness, sentience, creativity, ...). This reveals that there is a necessary metaphysical component to those attributes, at which point any scientific-minded person will leave the debate.
OK, I see, thanks. That applies to some skeptics, though - I'm in the AI skeptic camp but only in the sense that I believe we should make sober assessments of what these things are actually capable of.
> If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
To me, the goalposts were already defined by the person you were responding to. "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
Note that this does not depend on things like AI sentience or defining "intelligence" more rigorously, it just depends on AI impact.
> "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
those are not goalposts - that's an incredible vague 'goal'. What are we moving here exactly?
The "goal" here, not in the sense of what AI scientists are shooting for, but in terms of "goal", "goalpost", and the idiom of "moving the goalposts", would be to invalidate the current argument. Typically, you "goal" when you successfully disprove the other person's argument.
Thus "moving the goalposts" is for the defenders to shift what argument that the other side is trying to disprove.
If the argument is "the impact of AI is getting undeniable", you'd try to show that impact of AI is not that high, and not looking to be that high in the near future.
The other side might move the goalposts by saying something like "well, it doesn't matter if AI is actually that broadly impactful right now, all that matters is whether it's impactful in the field of mathematics". At which point, you might be frustrated because the effort you put into engaging their previous argument is now moot.
The unified position which many folks deny is "AI is powerful" or, alternatively, "AI will be powerful soon".
(I'm personally still skeptical about this, but I'm being pulled towards accepting it).
"AI is useful" is too low of a bar, and "singularity is nigh" is too high. "AI is on its way to upending society" is about in the middle, and still vastly contentious among laypeople.
I think Ed Zitron is mentioned just because his name is Zitron. He just repeats ad nauseam opinions concentrated around one simple, very boring pole on a wild and interesting landscape of emerging reality. Anybody could be doing that. A lot of people do that. Yet no other is named Zitron. And that's why I heard name Ed Zitron hundred times. That's how you become a voice of (a part of) the generation. Just have a memorable name and repeat the same opinion over and over that people can flock around comfortably. Content is irrelevant.
Personally I prefer to follow explorers rather than swamp-sitters.
"it isn't clear whether generative AI actually provides much business value at all"
"cannot seem to find a product that people will pay for, in part because the results are so mediocre"
"Last week, we got our first real, definitive glimpse of what’s around that corner that future. And boy, was it underwhelming."
"OpenAI claims that o1 “performs similarly to PhD students on challenging benchmark tasks in physics, chemistry, and biology.” Just not in geography, it seems. Or basic elementary-level English language tests. Or math. Or programming. "
"Worse still, it's kind of hard to explain why anybody should give a shit about o1."
"o1 shows that OpenAI is both desperate and out of ideas."
"the software is not becoming more useful"
Honestly, every other line is quotable in this context.
Yep, he is a PR stunt guy, and the number of videos that come up when you type Ed Zitron into YouTube should tell you how many people are eager to feed their cognitive biases.
They aren't claiming that science doesn't progress by moving goal posts. They're talking about how critics of AI have claimed it isn't revolutionary/useful, then progressively changed what would it mean for AI to be actually revolutionary/useful.
Not long ago many folks were saying AI was the same as the crypto bubble. No real useful technology and only hype.
> Never understood all this talk about moving goalposts - you understand that's how science works, right?
I agree with the parent that we need to acknowledge that we're at a turning point in history. I lived through some of them (internet, ubiquitous personal computing). But it's somewhat difficult to comprehend the impact of this one for many people.
I do biomedical research at one of the top European research institutions. We're very well-funded, but I can clearly see the gap between us (say, top-100) and top-10. I also realize this gap is going to get so much wider unless we invest heavily in AI access (and I'm not so sure I can sell anything more expensive than $20 Claude subscription to the leadership).
I think people having 6-7 figure SOTA AI budgets will move exponentially faster than those who don't. That makes me worried.
So, for me, it's not a question of recalibrating expectations. We're way past that.
People will be broken out of their denial by actual economic growth. That's what this is all meant to be for... I think we might start seeing some surprising numbers.
Plenty of people are making money from AI? And I am not sure I've seen it "making things worse for most people" yet, do you have a source behind this?
If your source is AI layoffs, there were plenty of layoffs with the invention of the horseless carriage, but that doesn't mean it made humans worse off overall.
There are reasons to be skeptical about progress and AI and all that but the 'making things worse for most people' thing you mentioned seems yet to be based in any reality
most devs i talk to say that they would love to go back to coding without ai, if everyone had to do it, so there's that. Oh also scammers have gotten so much better with llms, so a significant amount of peoples lives are worse due to that
Is the amount of people making money from AI greater than the amount of people who lose money from AI? Is maximizing prosperity across humanity even a goal, at this point? Am I to be expected to believe, without doubting, that the end goal is the benefit of the many?
But who benefits from that economic growth? If it's only the capital owning class, and the rest of us are left in the dust, then the growth is irrelevant.
I ,for one, have read enough history to know that it's never the proles who end up benefiting.
I suspect that you have not read enough them. Ask someone in China (which yes, still has a capita owning class) if the proles were better off 50 years ago or now. The only real probes we have now in the west is that we have failed at housing, precisely because instead of growth, we have made many choices to increase returns for old people, and insufficient taxes for real estate.
I was a bit triggered by your China comment, if only given the fact that there is a very specific institution that drove many formerly prosperous cities and communities into abject poverty (the CCP in Mao's era) and then tries to claim all the credit for the market driving how many people they've lifted out of poverty.
And nowadays with Hong Kong (and probably soon Taiwan) they are proving they are perfectly happy to destroy economic growth as long as it benefits The Party
Can someone explain why we’re not using it to solve the obvious big conjectures/problems though? If this stuff is solved, why isn’t Riemann the first thing to go? Even if it means a fund of several hundred $k to let it churn on it.
Guys, please use critical thinking. The haters don't hate by default, we hate because we're gaslit about this stuff every day and it's annoying. Extraordinary claims require proof, and they're not giving us information that would be essential to knowing if this is actually significant or not.
AI already have an impact, and yes this is PR hype because this is a product. Yet both can be true at the same time. We're not blindly eating what's OpenAI is serving us as gold truth, we're just admitting it's doing remarkable progress.
Remember October 2024 Pelicans [1] ? It's been only less than 2 years.
We don't know what will come in the next 2 years. But the progress doesn't seem to stop for now.
People are skeptical of the announcement because the room include several PHDs in math and physics. The prompts are not published so we can see how generic the starting prompt is.
Nobody is saying it doesn't have an impact. What we're saying is the near-religious fervor isn't warranted. AI boosters always speak in the future tense, which is extremely telling because we have right now is basically "ok" not earth shattering.
If you think it's "ok", then you're probably not using it the right way.
I've been analyzing thousands of x86 traces and the AI made sense of them at a scale no previous human or algorithm could do before - I've been reading assembly everyday for half of my life. I'm doing prototypes in minutes what would take hours or days before. AI is answering questions in such a precise way in so many domains, learning something has never been easier.
This is "just ok" for you ? What do you need exactly ? god omniscience ? come on, can't we just appreciate for what it is and be excited about the future ?
Gary doesn't argue it's hype though. He argues 2 things: other people are getting carried away with the result, and we don't know enough about how it was reached to know where it falls on the impressive scale.
He literally says it's an impressive feat in the second article.
best point in that first link: openai doesn't tell us if they only tried to solve these 10 and each was solved (amazing) or if they asked it to solve a million problems and it got these 10. Either is great, one is more so.
Gary Marcus has been moving the goalposts since day 1. The guy is a psychologist. Why would anyone care what a psychologist has to say about AI? He's likely made good money from constantly moving the goalposts and being a denier, due to the publicity he gets.
He's simply a good phone number to have for journalists under time pressure who need to add the contrarian voice to their upcoming story. He delivers it reliably, then never reflects on how he was wrong in the past, just blasts forward as if nothing happened and just makes the next bonkers claims to the journalists who are very thankful for the prompt delivery of how AI is a nothingburger, and fake and won't ever do XYZ that it then proceeds to do in N months.
I remember the time when he insisted that diffusion-based image generators trained on Internet scale data will never be able to make an image of a horse riding an astronaut. Today you can generate 4K video of that.
Mate, if you don't like Gary Marcus that's your call, but you're completely just lying when you talk about his qualifications. He is not "just" a psychologist, he's done a lot of AI research and even started AI companies that were acquired. This is absolutely a person with the credentials to speak on this subject, and honestly all his takes I've read have probably been TOO nice to AI companies.
Your comment seems entirely disconnected from the posts you linked. It's impossible to deny the results, it is not PR hype that in the past couple of weeks LLMs have resolved problems that have been open in mathematics for many years. Some of those problems had remained unresolved despite keen interest from many humans.
The only way that is PR hype is if you're invoking the insane conspiracy that frontier AI labs are just buying off results that would otherwise be career defining for a mathematician, just for marketing.
The posts you linked are urging caution regarding the exaggerated e/acc-esque lies peddled by people like Musk, not that the models haven't proven themselves as having genuine ability to contribute to research in some areas.
I use these strange machines all the time. They have gotten notably smarter. That’s my personal experience.
They still do things that I find incredibly annoying and “dumb”. And I still have to clean up messes they make quite often.
But on the whole they are clearly smarter than before. No extraordinary claims needed. I just try to learn how the tool works and how to use it effectively.
In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
How does intellectual property work in an AI generated future? What about healthcare advances, who gets to own those?
What about meaning, what about purpose? How do we replace the work ethic that tells us we are our jobs and idleness is immoral? How do you replace “What do you do?” As one of the first questions you ask a new person?
I think this too is a kind of denial. As in, while some are in denial about the usefulness of AI, others are in denial about whose living standards are actually going to be uplifted.
And it's sad, really, because I think these two groups would make a great pairing if they could stop arguing against one another for a moment. They'll both be impacted about as much and probably have the same ultimate goals (to lead dignified lives).
But it seems these days everyone is more interested in Kayfabe and feeling like they're in the right than working together, so maybe I should just keep quiet rather than attract the ire of both groups...
> others are in denial about whose living standards are actually going to be uplifted.
I don't know if it is fair to say they're in denial. For my part, I don't expect life to get much better for regular people (especially short term), but that doesn't mean we shouldn't work to try to make it happen.
Asking questions about policy and values and pushing to have those resolved in positive ways is about as far away from denial as you can get. It’s possibly the only useful thing an ordinary person can do.
What a lot of people want to do, and I’m not saying that you’re one of them, is to assume that a positive outcome is impossible and either do nothing or loudly yell that the world is ending. Neither is particularly useful.
Or, as I said above, others just deny that there’s anything to see here and try to get people to move along.
The AI companies themselves, who are highly motivated to sell AI as overall positive for society, notwithstanding some security/etc risks, and who have economists on the payroll to think about things like this, do not seem to have found any possible positive outcome to present.
Shane Legg (DeepMind co-founder), one of the more intelligent and thoughtful people you'll find in the industry, could only offer "it's a tough problem - we need to think about it" when recently interviewed by Hannah Fry.
On the surface the most likely outcome for AI allowed to replace jobs is extraordinarily negative, especially since it is a general capability technology, not a specific one where displaced workers can just move to another field. Once AI becomes more capable it will be able to do the vast majority of white collar jobs, including any new ones that may appear as a result of AI. As Shane Legg put it, "if your job can be done remotely, sitting in front of a computer, then it can probably be replaced by AI".
Not only does AI threaten to replace ALL the white collar jobs, but it is rapidly going after blue collar (factory jobs, driving jobs) and pink collar ones (Japanese robotics for elder-care) as well.
If a positive outcome (which doesn't include putting displaced workers on welfare - UBI) is possible, then it sure would be nice to hear it, and the silence from the AI companies, and government for that matter, is deafening.
UBI probably is the positive outcome, though it may not seem like it to begin with. Initially it will likely be stigmatised and under-resourced, but as a larger proportion of people move out of work and onto UBI that stigma will drop and the resources should grow.
Eventually UBI will be the norm, and if the living standards of a person on UBI is as good as yours or mine today, that will be an enormous win for everyone. It’s like pensions, once these were only for the elderly poor, now they’re a right for everyone in most developed countries.
It’s also interesting that for most of human history leisure time was the point of life, and only in recent modernity has work come to be the meaning of someone’s existence.
UBI has to be commensurate with production being automated. That’s a big logistical problem, if you think building datacenters is a challenge try bringing about radical abundance, but even so it’s not insurmountable. It just needs to be taken on as project and not seen as an impossibility.
So much of this is not about what is possible so much as what people believe is possible. We can do anything if we try.
We don't see a progress towards UBI even inside individual countries and with AI we will need it on international. What people in Europe or Africa will do when all jobs there will be replaced by American or Chinese AI companies, where they will get money for UBI?
No, I think people will demand to not live in poverty. Capitalism allows people to escape poverty by personal effort. An AI future won’t allow it by any means other than collective effort, so that’s what we’ll do.
That will entail an immense amount of violence to be undertaken by the working class against those in power, unless you think they'll gladly give up all of their ill gotten gains out of the goodness of their black little hearts.
In fact I don't think that rich people are cartoon villains who enjoy watching the poors starve. If AI leads to greatly increased productivity, then at existing tax rates there will be more than enough to provide good living conditions for people who can't find jobs.
In some countries, especially less developed ones, poverty levels are declining, but in the US the trend seems to be the opposite. The middle class is being gutted and descending into survival mode if not poverty, while the top few percent become obscenely rich at their expense.
In any case historical trends are really irrelevant to the discussion, which is about the impact of AI, which threatens to take away almost ALL the jobs (this is what Dario's essay is assuming), which has no historical precedent. Some people like to bring up previous waves of job automation such as the industrial revolution, but the difference there is that automation took some jobs but created others. In the case of AI, AI will also be taking the new jobs that it creates.
If AI takes all the jobs - which is what the people like Dario who are creating it assume will happen - then it seems "UBI" is indeed the logical conclusion (other than those able to make a living working for themself, or via self-sufficiency), but this is not going to be utopia where we are all idle rich practicing our hobbies. What it really means is a welfare state, where formerly proud people capable of supporting themself become dependent on government handouts. What comes to mind is the movie "Soylent Green", not utopia.
I'm not sure how anyone imagines this would actually work. Is there any private enterprise left at all, or are all the means of production (AI datacenters and robotic factories) all controlled by the state. Instead of distributing Soylent Green, the state distributes UBI-scrip, essentially food-stamps, that can be exchanged at state stores for provisions?
Seriously, how could this actually work ?
An alternate future, no more optimistic, at least in the short term, but perhaps more realistic, is that in the fairly near future when unemployment and public pain becomes high enough, we reach a tipping point, and the pitchforks come out. Eventually the government concedes that AI is no more conducive to the public good than nuclear weapons, and heavily regulates it, banning the use of AI to replace jobs. This may sound extreme, but surely not a fraction as extreme at the UBI-based welfare state that Dario Amodei is fantasizing about as the best possible outcome (that is compatible with himself becoming enormously wealthy).
> No, I think people will demand to not live in poverty
What about other countries in the world where people... live in poverty?
> Capitalism allows people to escape poverty by personal effort
Isn't it possible that there is a future where AI makes the average value of human labor (or, "personal effort") plummet? Perhaps capitalism will lose some of its edge against a technology like this.
This seems more a fantasy than a considered likely outcome. He basically admits that humans will eventually mostly all be out of a job, replaced by AI, but then says (Gemini's summary) that there will be:
"Massive Economic Abundance: Because AI will exponentially grow the total economic pie, overall resource scarcity will diminish. The fundamental challenge shifts from producing wealth to distributing wealth."
So how do we go from everyone out of work, no income to spend on food, or the goods and services that the AI is producing, to "massive economic abundance"?!
"Massive Economic Abundance" implies massive increase in produced goods. This implies massive deflation, ceteris paribus. So step 3 could be simply printing money to pay for UBI. Deflation from AI productivity increase and inflation from UBI money printing will cancel out.
For money to work it has to represent some real value, something that has some scarcity to it such as potatoes or hours of human labor. Ultimately it is just a decoupler in a barter system, a universally recognized IOU.
Why would someone give me a car in exchange for UBI-scrip when that UBI-scrip has no inherent scarcity or value and can be produced in infinite supply by the government ?
> Why would someone give me a car in exchange for UBI-scrip when that UBI-scrip has no inherent scarcity or value and can be produced in infinite supply by the government ?
UBI script will have some value, I am proposing printing enough money just to combat AI productivity-induced deflation, not infinite UBI money.
You know, I think Dario should have kept the Richard Brautigan theme in his essay titles. I think that I was trying to describe you to someone is an absolutely perfect match for introducing an AGI that has been achieved. It’s basically the same idea in blank verse, about a spread of a different technology, and made personal along the way.
Asking politely is not how we got a 40-hour work week or workers' comp or most other labor standards we take for granted, just historically speaking. I think a positive outcome is very likely, and I think it will be a lot more work than loudly yelling, but I don't think anything will happen if we try to build everything up from first principles instead of taking a moment to consult history as many are wont to do in this AI era.
What pressure is there to push for any of these changes? I see AI advocates discussing the concept of UBI, but I can't imagine a world where the United States would ever pass this sort of legislation. I mean, congress can barely pass a budget each year.
If you are correct, I expect corporations to reap massive profits while most Americans try to find a way to survive in a world where they are obsolete.
There is whatever pressure you bring to bear on it. As long as you are still living in a democracy your vote is the pressure you can apply. If the parties that exist won’t represent you, make new ones that will. Do something rather than deciding it’s both impossible and up to other people anyway.
I also don’t understand why UBI is desirable. Putting everyone on welfare means everyone is poor. This won’t end well, and it’s certainly not the case that 99% of the population will let a tiny number of people remove their income in favor of pennies for all. Political violence will come first, easily.
Because the implied alternative of the comment is that people will carry on having jobs, which breaks the entire premise that there is AI that replaces all human labour. The comment is missing the point - UBI is the only suggestion anyone has so far that meets this fundamental question about the technology. We will be less poor than without UBI, unless we come up with a better plan.
Isn't it obvious if AI automates your job away, the AI company is going to reap the economic value, which is going to be less than what you cost, while giving you a pittance as UBI? If you received its full value there wouldn't be any point in anybody replacing you with AI.
Agreed, a LOT of UBI advocates gloss over the "B" in UBI. If AI increases human productivity overall, the only morally acceptable outcomes (imo) are that everyone's standard of living increases (or at least is the same without having to work) and wealth inequality decreases (if AI is doing ~all the work, there really isn't any sensible justification for some people having significantly more wealth than others). Frankly anything else seems like a recipe for massive social instability.
> In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
> We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
UBI in the United States is never going to happen in time. If it happens at all. We don’t even get universal healthcare. I think people who think AI will be a net positive for humanity are also in some sort of denial.
In a different US political climate I would entertain it. If these frontier labs weren’t so clearly going after the money, I would entertain it.
LLMs are clearly a step up for capitalists so I just can’t see any inclusion of LLMs move towards more progressive ideologies.
I guess we'll have to fall back on the plan of hoping one of the 7 Anthropic co-founders has secure power in this world, and decides to use EA principles to give us all a bit of their share of the future light cone ;) (The wink because I'm kinda not really joking that this seems the best plan, if we actually get superintelligence from a US AI company)
Everything you're saying is oddly simultaneously specific and hand-wavy at the same time. I say this because I'm not sure a lot of these things are solvable or, if they are, it's going to take lifetimes. For example:
> How do we replace the work ethic that tells us we are our jobs and idleness is immoral?
For many people it has nothing to do with morality, it's hardwired into their instincts. They want to work, and they will work.
For a lot of us who are not excited about this future it's that no one is trying to answer all the questions you laid out. Instead we have the disgusting people at the helm purposefully spreading doomerism and saying, "We'll figure it out." I think it's pretty problematic (to say the least) to care more about technological advancement than how that advancement is actually shaping up to effect people in the short term. But I know many people don't care, especially those who believe they won't be among the affected.
Beautifully put. This exactly summarizes my feelings about this particular tech. I actually find it hugely useful. But I fear we lack the wisdom to use this in a way that won't destroy us.
I don't understand why this got downvotes - simply extrapolating current trends leads to the need to answer all of these questions.
My own $0.02 on the economics piece - every country should have a sovereign wealth fund. Governments should block market access from automated[0] companies until those companies provide equity contributions to the wealth fund for that country. This aligns regulator and corporate interests. Dividends flow into the sovereign wealth funds and then can be allocated locally from there - UBI, job programs, etc. Let different jurisdictions explore different ways to structure a post-labor society.
On the broader social front - I think a lot of lack of meaning discussion boils down to the overemphasis we have on your job as your self-worth. We need to realign our societal expectations - and people need to spend more time with their families.
[0] for this to work, I think we would need well accepted metrics for 'how automated' a company is - and that probably needs a 3rd party auditing industry.
Nationalize the data centers, reserve enough inference to automate power generation, food production, transportation, and housing.
Is there any government that has gotten socialism correct for its citizens? I'd point to UAE/Qatar if they didn't depend on human servitude and inequality.
Why do you think the AI companies are going to let it be a “we” kind of decision, and since it’s obviously not going to be a “we” kind of decision, as getting to this point certainly has not been thanks to people like you, what makes you think living standards are going to be broadly uplifted?
The human zoo where the top ~250,000k humans live in a "human utopia" and the AI provides while mostly focusing on whatever it decides it's own goals are.
Humanity survives (but we reading this probably don't), the AI treats the living humans like the Emperor's favorite pets (probably a pretty good life), and then the AI does whatever else it deems important.
Realistically it means trying to start a small business of some sort, because AI is hugely advantageous to business owners and disadvantageous to workers. And it's something AIs can't do unless they get legal personhood, which may well not happen any time soon.
Hopefully doesn’t ever happen, since that’s one of the more plausible omnicide scenarios. I human like AI is almost certainly achievable without much research effort at this point, but we shouldn’t do it.
Logical mistake. There is a difference between every human dying eventually and humanity going extinct. The former doesn't imply the latter. The previous commenter clearly meant the latter.
Your 'serious' proposal is unlimited global military bombing campaign on civilian infrastructure by the United States (which is currently losing a war using the same strategy) to 'solve safety' preemptively against a 20% number you just made up?
And you accuse the 'other side' of 'suicidal apathy'??
You should put down the AI and do some self-reflection on how you came to hold these views.
> You should put down the AI and do some self-reflection on how you came to hold these views.
I've been in the field for almost 2 decades, and actively thinking about AI risk for over 15 years.
I've always held the controversial opinion that there may come a time where we might unfortunately have to consider using force to protect our civilisation from the threat of ASI, but I've always reframed from openly discussing that because I'm so personally against any use of force and there's always been time for more peaceful options.
Several months ago I changed my tone on AI risk to stop worrying about the optics of what I'm saying because it's too late in the day for me not to speak plainly at this point. Similarly over the last week I've decided I can no longer reframe from advocating for the use of force (if necessary) to stop rouge actors from playing Russian roulette with civilisation.
Please understand I don't want anyone to be harmed. Perhaps you don't agree with my suggestion, but my stance ultimately comes from a position of harm reduction.
> preemptively against a 20% number you just made up?
It's a probability estimate. Happy to expand in detail on why 20% specifically, but first you accept I can't tell you what's going to happen in the future with certainty right? To some degree any prediction you might have about the future will be "made up".
Advocating for nuclear war as a means of 'harm reduction' should be causing you serious cognitive dissonance.
There are dozens of escalating, yet fundamentally less destructive means to prevent a data centers from continuing to operate (if that's the goal). Not starting at those more modest interventions is political malpractice and in opposition to your harm reduction narrative.
Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.
Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.
Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.
> Advocating for nuclear war as a means of 'harm reduction' should be causing you serious cognitive dissonance.
If you're referring to the "nuke the data centers" line in my bio, that's tongue in cheek. It's an expression of my sentiment towards AI at the extreme.
To be clear, I don't think any country should seriously be considering dropping nukes on data centers. Targeted strikes on data centers operated by rogue private companies or nation states is acceptable to me – assuming pre-warning of intent and appropriate regulatory foreshadowing.
I suspect we agree this is reasonable?
> Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.
> Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.
No, they're not based on unbounded exponential growth curves at all... I'm actually somewhat skeptical of claims that there could exist an intelligence that far exceeds our own. My assumption is really just that AI will be far faster, far more knowledgable, far more persistent and marginally more generally intelligent than the average human. The first three are already true, the forth we're getting to.
We're on an exponential curve right now and I suspect the rate of progress will accelerate some more as a result of RSI. This will likely continue for 2-3 years (if we have that long) before starting to flatten out.
As for my doom prediction, my median concerns are around around the destabilising nature of ASI.
If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation. I can go into more detail here if you wish.
However, I'd still put these destabilisation risks at under 50% in total. My concern more broadly is the majority of ASI outcomes are doom scenarios. It's extremely hard to imagine a world in which you have a magic box that can grant any wish that intelligence can grant, but which we have a functional civilisation. You have to basically assume all potentially dangerous wishes won't be granted and probability the only feasible way to do that is restrict who can make the wishes – but then you have the concentrate of power risk.
I largely agree with the premise "If Anyone Builds It, Everyone Dies", so we must try to avoid building it. I think we can build ASI safely, but we're clearly not on the path right now and I suspect it might take decades (or more) to figure out how we can build ASI with an extremely high probability of a good outcome.
> Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.
I do this. I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.
I don't think we realistically have a chance of diverting from this path at this point. I have made peace with this but it makes me very sad and haunts my dreams almost every night.
> If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation.
We already live in a reality with unprecedented concentration of wealth and power in the hands of the very few.
This process has produced horrific violence and ended civilization for tens of millions.
The actual impact of AI, so far, has been redistributive upwards (solely through speculation, not via productivity or displacement).
This kind of problem is a standard human social ill. Its solution is social too.
Redistributive economic and political action can equalize power and wealth disparity. AI is incidental in this story.
Your understanding of AI as an isolated perfected intellectual djinn lacks a consideration of the existing and ever-changing labour and social relations that give AI responses meaning in the first place.
It is a compressed, multi dimensional manifold containing the structure of relations between objects.
Responses are a reification of a point in that space.
Navigating the space can be interesting, but it is a poor substitute for the relations themselves.
Very few people yearn for the endless, frictionless solipsism silicon valley promises (AI + slave robots), and those that do are overrepresented in your social circles. The books they write are a study in psychological projection and limited political education.
> I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.
This is good advice generally. But it isn't organizing. Join something, meet up, drink bad coffee, listen, take notes, advocate to fix just one bad thing in your community, feel your impact reverberate through the world, and then notice how much easier it is to shake off despair.
If you don't have a community, then no wonder everything feels like its ending.
If the thesis that "If anybody builds it, everyone dies" is true, or has any chance of becoming true, then it is the logical thing to do. As Geoffrey Hinton said, "If you want to know what it's like not to be the apex intelligence, ask a chicken."
It is not 'logical' to imagine a hypothetical doomsday scenario that justifies a preemptive nuclear war. (which is what the grandparent commenter's bio contemplates, 'nuke the datacenters' is their central credo).
Does it bother you that the people who are publicly cocksure that P(doom) is moments away are the same people that have profited most handsomely from that pronouncement?
That the 'humanists' that want to do 'altruism' for 'potential future humans' and are the same people that commit fraud and theft at a civilizational scale, then sell this 'intelligence' to any child-incinerating militaries with spare cash?
It's not wrong to want to do good, but if a system that is branded 'do (the most) good' commits great evils, you are morally and intellectually obligated to step back and reconsider how you are spending your time.
Also, I asked a chicken and a feral rock dove what it's like to be not be 'apex' and they burbled at me and kept eating millet and sunflower seeds.
Would you like me to follow up with them? I'm not sure what point you expected them to make.
I've removed the "nuke the data centers" line now.
It's was an expression of my sentiment, not a policy position I'd support. I wrongly assumed that was obvious, but some people are making bad-faith assumptions about me and my sanity.
Not that it should need to be said on a comment thread where I am express concern about civilisation in a post-ASI world, but I obviously don't want to see a nuclear war.
If the Judeo-Bolshevist theses were true, then Operation Barbarossa was the logical thing to do, and everything which came with it.
Powerful word, `if`. "You're not only wrong you're a fulminating psychopath" is a perfectly valid response to getting it wrong like a fulminating psychopath.
AI is a math & computer science problem. If AI can do advanced maths and computer science research, then it can begin to suggest useful algorithmic optimisations.
Right now I'm sure the vast majority of these will be junk, but occasionally, even with current limitations, they might occasionally stumble on something.
It's not really whether RSI is or isn't possible, it's really just whether it's the most efficient way for labs to improve their models today given they have limited compute to run AI-generated experiments and access to very intelligent humans who might have a better hit/miss ratio.
Do you disagree with anything I'm saying here? Do you not think current AIs can suggest algorithmic improvements or something?
A few years ago the majority of Hacker News dismissed LLMs as not much more than stochastic parrots who were decades away from doing any serious work, autonomously escaping containment and hacking Hugging Face, or outperforming most professional mathematicians at proving theorems. These things were labeled "insane" and "science fiction" despite the rapid progress we had already seen.
Now you are again postulating that there would be no more extreme progress in the near future. That's actually more "insane".
Not satire, but as you can see one is engaging in good faith and everyone is either engaging in character attacks or low-effort counters.
It's fine, and I'm very used to this, but if you want to have a dialogue I'd be very happy to. I'm a very reasonable and sane person outside of apparently holding some controversial views on AI risk =)
I'd love to know why you think what I said was insane?
I don't think this is a straw man, a huge number of people in my life (non CS people) think AI is a dead-end, that it's just a stochastic parrot, that it'll never be able to do many things that humans can do. I have had many arguments with people who told me that "AI will never be able to do X", and then 6 months later AI is able to do X. Then they will move the goal posts and say "well AI will definitely never be able to do Y".
I can deal with apathy, that’s the norm. What bothers me are all the people who think they can suppress AI by talking it down. That’s what’s counterproductive, just pretend the problem doesn’t exist. Tell other people it doesn’t exist either. I get it, it’s threatening socially, economically, maybe existentially. It’s also not going away.
So you think it’s a potentially existential threat but are bothered by people who maybe want to suppress it… Hopefully you acknowledge there is a bit of lack of self awareness here eh?
It is a fact of experience, and indeed effectively a theorem, that the better they get at coding and math, the dumber they are. These are the wages of RLVR etc
FWIW I was responding to the sarcastically expressed overt message that AI-math is a clear sign that AGI is here and anyone who disagrees is in denial.
My point being that AI math is a narrow skill just like AI chess and implies nothing about generality (AGI).
I agree. The most likely scenario is that this is just a "new normal" lift that is percolating through human endeavors and will saturate at some point. For example, the whole cyber-security bruhaha should ultimately resolve into higher standards for code published -- we can now cheaply find and fix a whole slew of minor bugs that weren't worth our time before.
The fact we see a lift is not the same as evidence that the lift is unbounded.
The lift being finite is supported by the fact improvements have come at the edges: improvements from human feedback, improvements in harnesses, improvements on model compatibility with harnesses, improvements in inference efficiency with new architectures, etc. If we were just training better models from scratch that would be one thing, but we are just making better use of a tool we've developed.
Not sure what your first sentence means, or why you are quoting AI. Many of these problems individually were math at a level approaching the highest possible for expert human mathematicians. These are not simple combinations of existing ideas, or following of human intuitions, or implementing something following specific human instructions. Then again maybe you mean that math is not knowledge and that all math is simply extending the basic axioms using known patterns, to which I would not agree.
as much as I love analogy with humans using super-shoes, not everybody can be a world-class expert in their industry. There should be a place at the table for average people to take part; otherwise, it won't be sustainable.
As a programmer, I am mostly interested in whether my role is sustainable long-term and whether the models will get better. I don't feel in jeopardy yet, but two more years like this and the calculus of hiring software engineers could shift even further. QAs are already overwhelmed with work
well not every runner can run world-class sub2 marathon or even vaguely close, not even in super-shoes
but with super-shoes more and more runners are qualifying for boston marathon and even olympic trials marathon where it would have been impossible for them previously
and that's what "AI" currently does, it allows average people to immediately "pick the brain" of every expert in every field, in every scientific paper, without previously reading a single other google result, something that would have been impossible for them previously (super-shoes for the brain? too far?)
but "AI" isn't creating new knowledge, it's just stitching together existing knowledge from patterns that would have taken years by human hand if even possible at all, it's going to "hit the wall" eventually (in its current form)
How do humans "invent knowledge"? Is your argument that the answer to these questions already existed in the training set? Why didn't any human recognize that before?
humans discover a lot of things by trial and error (aka what I meant by "invent knowledge")
basically everything Benjamin Franklin did was trial and error because no-one understood what electricity was in the slightest
almost everything Edison did was trial and error too, he had his lab try thousands of materials for his long lasting lightbulb filament
even the most advanced "AI" today is just machine-learning going through everything already known trying to piece together previously discovered facts, admittedly at levels and detail impossible by human hands
but that means there are limits and it's not really "AI"
Have you seen how it debugs? It's trial and error. As the most reasonable hypothesis fail, new data is found, and it keeps iterating. It's not a human intelligence, but if it has an issue, it's not some mysterious creativity anima in the heart of man.
I have seen it produce tentative genetics for experiments, just like a scientist does. Then the data comes in and it can evaluate the data from the experiment just as well. One just had to give it a lab budget.
Any computable problem will eventually fall to computers.
LLMs have made math proofs more computable, in the sense that a computer can both generate potential solutions and check the validity of its solutions on its own, with a reasonable chance of converging on something correct. I assume this was already doable to some extent, but it seems like it’s now exponentially easier. That still doesn’t mean that all math is automatically solved.
This is somewhat similar to things like molecular dynamics or protein folding or finite element simulations, etc. Some problems that were previously intractable via computation became tractable. Others - the vast majority of other problems - remain unsolvable by these computational techniques, because the scale of compute required is beyond imagination. These are simple things like simulating the dynamics of a cubic millimeter of water molecules for 1 second. Unfathomably beyond current capabilities (and LLMs aren’t going to change that).
I think LLMs are great, I use them every day and I think they have a ton of value. But if these things were as revolutionary as people promote/fear them to be, you should immediately point them at the highest value math problems and see progress. Like the Millenium Prize problems. Haven’t seen a solution to those.
So there are limits - but we’re about to learn a lot about the new normal of what constitutes a layup math proof vs the truly difficult.
> Any computable problem will eventually fall to computers.
I think the question, that we keep stumbling over, is what problems are computable.
> But if these things were as revolutionary as people promote/fear them to be, you should immediately point them at the highest value math problems and see progress. Like the Millenium Prize problems. Haven’t seen a solution to those.
Let the goalpost shifting continue. It'll buy us another half year or so.
If by solution you mean a proof and by testing you mean encoding it in lean and compiling it, the space of possible syntactically correct proofs which you can encode probably explodes in a way that is well beyond what any computer could try to brute-force. LLMs don't brute-force proofs, i believe their approach is quite similar to humans. I believe the same is essentially true for counter-examples of the type that have been found latelly, they are not found by search, but by using theory.
On the other hand even if the compute allocated by openai is esquivalent to day 10 human mathematicians, the machines can work 24h per day, that is already a lot more productive.
Whether or not it’s on par famously or difficulty level doesn’t predict whether the others will fall. They’re unique problems and math isn’t linear - the Jacobian it managed to find a counterexample and relied on other proofs that had been developed showing >3 case == 3 dimensional case. However, the Riemann may not fall in the same way because it may actually be true or the surrounding math isn’t quite ready to tackle that problem.
I would be genuinely very impressed - but still not scared - if the Riemann hypothesis were solved. I suspect that we may require “new math” to make progress on that. If a new operator / symbol is required, is that fundamentally not doable by an LLM because it’s outside of current tokenization space?
Whenever the handwaving starts around a discussion relating to a NP hard problem, I find it useful to imagine a Canadian bloke (MHRIP) in a red top, with a ... Scottish accent ... saying:
"Ye cannae break the laws o' physics, Jim". (maffs not fisics, obvs!)
If that is a bit tiresome for the gung-ho AI evangelist, there is also the rather knotty snag that that blasted Austrian geezer Gödel fiddled up: incompleteness.
Its almost as though these bloody clever scientific and that types keep on putting artificial blocks in the way of LLMs laying golden eggs!
I'm quite happy with the "marginal gains" I get with a DGX Spark. It will pay for itself within three months doing stuff on prem and us not sending data to someone else. It will scale.
Not being an expert in any of the fields OpenAI has "advanced" I don't want to prematurely downplay the significance of this contribution. However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing.
It is true there hasn't been a reliable computational approach to solving these problems before. But do these proofs contribute new ideas to the mathematical corpus, or are they simply an effective method to exhaustively search the literature for the right combination of existing tools to apply to the problem?
Essentially, did these problems seem like they had an intuitive answer and were feasible to prove before, just not high enough value targets for an expert to invest time into? Or were they fundamentally difficult prior to this point and it appears that AI has done something more than just throw the problem into a big solver.
The ones I'm familiar with are big breakthroughs, but they are both counterexamples. Examples have an advantage in that once you have the example in hand and a sketch of the proof (which they have provided), then an expert can probably work out the details themselves.
The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more.
The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.
> but proving a group was non-sofic was out of reach
a colleague was telling me that the base idea for proving that something is not sofic already appeared in the literature around 2019 or so (this is the "expanders graphs" that are mentioned in OpenAI s paper. no one had managed to find a concrete example though. this doesn't make the result less impressive in any case.
Interesting. Do you have any more specific insights into where you feel AI was a big value-add to these problems? I don't want to be overly dismissive of AI, but I also feel that the AI hype engine frequently positions claims as being 'ground-breaking' when they are really just interesting incremental results.
The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.
They are more than a strong student could achieve. I'm not equally familiar with the problems, but the ones I'm familiar with, if a student solved them people would be thinking "that's someone on track to win the Fields Medal one day".
If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."
The problems from CS (CVP and circuit complexity) are very important problems that have been worked on by top researchers for 30-40 years. Some of these researchers include Turing Award winners. A solution to them would be a best-paper award at many top CS conferences.
I assume you're talking about No. 5, the arithmetic circuit complexity bound? The existence of a lower bound than state-of-the-art is certainly a significant result and worth publishing.
But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).
Circuit complexity lower bounds (and lower bounds in general) are notoriously difficult to come across.
For example, despite our best efforts, the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n). In contrast, our best algorithms for the task run in time roughly O(2^n). That’s an exponential gap. This is despite decades of trying to find lower bounds.
> However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing
Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.
I have no idea how many PhD's have spent how much time of their careers tackling these very specific problems, and I doubt you do either.
I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.
They are significant problems which experts have spent months or years studying. I heard a mathematician say that resolving non-sofic groups and Connes's rigidity would be career-defining for a mathematician.
"Breakthrough research" can be defined (in the citation record) as research that both (1) becomes highly cited, and (2) brings together citation chains that were previously not showing up together.
Mundane incremental research is cobbled from existing citations that already appear nearby in the record.
Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record.
So we can know pretty conclusively.
Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on)
Breakthrough math research is very rarely highly cited. Maybe some combination of pretraining scale, inference speed and orchestration will help, but it's telling that OpenAI is solving random math research problems rather than bedrock algorithms and their implementation. Even as cool as the tech is, there still is very much a clock that they have to outrace before they collapse.
I'm not arguing that this isn't innovative or worthy of publication. Basically any result that moves the needle meets those criteria. I'm interested in how the results that OpenAI has published here differs from finding optimality solutions for incredibly niche optimization problems by throwing the problem in an enormous solver.
They differ in that there hasn't been a solver you could have thrown them at. I guess you could argue their harness + LLM setup is a "solver", but the approach is so different from what we used that word for in the past that I don't think it would be appropriate.
It’s a marketing. They are a sham company. If this article was by Scientific American or something it would be worth a lot more. They are literally trying to keep the hype train on track.
Also on HN front page today: AI's debt binge can't last, hidden borrowing reaches $1.65T (fortune.com)
It is marketing, they are a shady company, and yet, if someone had access to these results before today's modern AI tools, they could get tenure at any university in the world.
If the prompt was crafted by an expert mathematician with pages of context and insights provided to the LLM to put it on the right path, that is a lot less impressive than a prompt that just says "find a counterexample to this conjecture".
Similarly, if they are spending millions of dollars in inference just churning on thousands of problems and these are the 10 solutions they came up with, that would be less impressive than if they chose these 10 problems specifically and were able to come up with solutions.
one of the early premises of how ai takeoff would go was that a system that could solve open problems in advanced mathematics would also discover novel advances in math and computer science that directly unlock drastically better software performance.
we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B).
we are also seeing incredible advances in software performance. open ai announced like 15% improvement by fixing gpu kernel issues.
these are clearly linked in the sense of scaling laws and generalization of intelligence: a huge model gets capabilities in both math and software engineering that isn't possible at smaller scales.
but it seems less likely to me than before that the types of math/science discoveries will explicitly unlock better software performance. in some sense this fits our intuitions. when top tech companies use math PhD type employees, they have them stop doing pure math research and instead focus on software engineering. these people are often very good at software engineering but not due to recent discoveries in academic mathematics, it's due to their general intelligence.
to me, this is evidence that the models are getting better but does not make me think we are on the cusp of a foom style fast takeoff enabled by revolutions in frontier math
(i also posted this on twitter @mlipman13)
I don’t really like AI but let’s stop kidding ourselves, no human mathematician could make progress on a dozen major open problems in a week or two. If you’re measuring it against humans then it is by far the best mathematician to ever live.
I guess it depends on how to measure a single "person"? If you spun up 2000 copies of Terrance Tao, I wouldn't be surprised if you found a few new discoveries at the end of it.
> I don’t really like AI but let’s stop kidding ourselves
If I had to create a tagline to describe my opinions about AI in a single sentence, that’d be it.
It’s possible to both hate AI and be impressed by it at the same time. Lying to ourselves about its capabilities does us no good. It’s emotionally difficult to do, but people need to come to grips with what’s happening and shake themselves out of a state of denial.
How much investment has gone into OpenAI versus mathematics research in 2025 for example? Probably 100x?
The AI results are clearly impressive. But these sorts of things are also in the ballpark of what human effort could solve given enough attention and time. Though it is hard to say.
This makes no sense. To believe this you have to think that the models are somehow being overfit explicitly on academic mathematics and it doesn’t carry over at all to more practical software engineering. I wouldn’t make that bet.
For example, pi can be computed to arbitrarily many digits, but could only even in principle be accurately represented with physical objects to a precision that many humans could memorize easily. This is thanks to physical constraints such as "diameter of the observable universe" and "Planck length", at a minimum.
Even if pi is an approximation, it is still representable mathematically. Floating-point arithmetic is an algebra (in this case, a magma [1]), which can be studied mathematically.
Some of the problems solved here, at least in CS, have been open for decades, and have been worked on by very smart leading researchers in the field, including Turing Award winners.
Like, these would be best-paper awards at many top CS conferences.
Or, that the mathematical formalisms that model the limits of software performance are firm enough that barring P==NP, nothing much will change despite proofs of beautiful math.
15% improvements are usually called “fixing a mistake in the code” or “getting to that task in the backlog for optimizing that code we had to ship on a deadline”. They’re more likely the bigger the company: more contributors working in disparate areas means more low hanging fruit is probably lying around.
This is untrue. It is very ordinary. How much do you know about GPU drivers that you state this so assuredly?
Drivers are software. Software can be improved. Do you believe there are no prior examples of GPU drivers being improved such that particular compute patterns go up in performance by more than 15%?
This driver improved Total War performance by 71% https://www.nvidia.com/download/driverResults.aspx/74714/en-...
Also note you can go ahead and improve any open source driver right now, most likely. Compile it for your specific card and remove all other architecture specific if-cases and you can get an improvement.
Correct me if I'm wrong, but all of the aforementioned advances were made in the last year? Until very recently few people had access to these tools. Most people still don't know how to use ChatGPT, and very few use tools like CC regularily. If in a few years these frontier tools become commonplace and people upskill we would should see a network effect?
> but it seems less likely to me than before that the types of math/science discoveries will explicitly unlock better software performance.
You seem to overlook a simpler barrier. To make these advances, they have to be possible. A 15% improvement in GPU kernels doesn't evidence that significantly more improvement has been left on the table.
Google has also invested a lot of time into developing new hardware and new algorithms with AI (other types of AI, not LLMs). I don't know if it's paying off (haven't followed it closely) but they seem to think it's worth the effort.
This will depend on the problem; I expect big algorithmic performance improvements in AI since the algorithms are still new, inefficent, and constantly being improved. But maybe not for sorting, fast fourier transforms, or other well-studied basic algorithms?
A lot of algorithmic improvement in AI is ultimately bottlenecked by compute. It is very easy to come up with ideas that could improve models! But to prove that they do, especially at scale, is expensive and takes a long time.
> we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B)
i think you have misunderstanding of what mathematicians do
> i think you have misunderstanding of what mathematicians do
They get to make cool 3D plot visualizations of functions so obscure to me that they’re named after someone who is still alive - and/or get to work on cryptography for the NSA - I think?
It seems that a lot of folks misunderstand the guarantees that lean provides.
I just want to state that having "lean proofs" that build (checks) does not mean the actual real theorems we care about hold. Ignoring lean kernel bugs, ultimately a human (not an agent) has to verify the lean encoded theorem statements (specs/specifications) that the lean proofs are checked against. For non-trivial theorems such as these, this is an arduous and tricky task where even a little mistake could be fatal. AI generated lean encoded theorems can be huge and difficult to understand. I wonder if anyone reputable has audited these specifications.
In some cases you're right, but I think that's often a symptom of mathematics in Lean being relatively immature (i.e., it will get much easier with time). Even then, verifying the statement in Lean is correct is still much easier than verifying the natural language proof is correct.
Care to elaborate? Curious about this. Is this because LLMs have been geared towards understanding user user intent behind a prompt rather than following the instructions exactly?
Isn’t that a huge simplification? Of course the way you phrase the prompt can carry semantic meaning, maybe subtly, but still. And sometimes that matters a little and sometimes a lot. I’ve stopped numerous agent sessions over the last few weeks to reword my initial prompt to get the agent off an unintended track.
It sounds like he hasn't verified the results of a problem that he has personally worked on, so how many of these problems have actually been verified?
Besides for what others have mentioned, the lean proof could be proving something else. Given AI’s propensity to hallucinate, seems like someone should check the lean proof actually expresses what it’s claimed to.
Incorrect. The statement in Lean can itself be wrong. Moreover, they could be exploiting a kernel bug in Lean, of which we had one published literally a week ago.
Can’t wait for this stuff to have quality of life increases for the average person. So far all I see is that AI has made owning a computer more expensive, made some jobs redundant, increased spam and distrust with questionable authenticity of content and of course made some Americans very rich.
Not everyone works for Evil Corp. I work in the public sector and my work supports public health and safety initiatives. AI has allowed my team to get much more done than we would have otherwise which improves the quality of life of the people in my community.
So I would like to counter your cynicism with a “YMMV” depending on who you work for.
Are you just automating lab reporting and results faster? That doesn’t exactly improve the health of others, a faster lab does not cure an ailment or provide a better cure.
Like cool my lung xray only took minutes to determine if I have a lesion instead of a week or a few days, but I still have cancer.
Faster scans means more scans. More scans means getting scans earlier and tracking abnormalities on scans over time. This can lead to earlier intervention, which means you get treatment for the cancer when it’s stage 2 instead of stage 3 or 4, which maybe is the difference between living a full life and dying young.
It has had significant quality of life increases for me. I use LLMs for everything from:
- travel and restaurant recommendations. my last few outings have been entirely LLM-advised and they turned out excellent. LLMs seem to have ingested every single Google review, photo, and menu of every business on Earth and can answer very nuanced questions like "is the garlic chicken at <restaurant, city> garnished with coriander?"
- fitness, nutrition, accounting, therapy, medical, legal, immigration advice (sure it's not a real professional but you know what, it's pretty fucking close, and any capability gap is made up by having perfect two-way communication which you don't get when talking with a human)
- coding (work, side projects, personal tools, documentation & pricing questions, "review this code", etc).
- I start reading most articles with the prompt "Summarize this article: <url>". I just started a non-fiction book by pasting into Claude: "There are 12 chapters in the book <book-name>. Can you give me a 2 sentence synopsis of each chapter?". It reduces the "activation energy" hump and screens if it's worth reading at all.
- I use the LLM in my Tesla for on-the-fly advice for parking and other things. You can simply ask "what's the best Boba place around here?" and it will give you a decent recommendation. You can also follow up with "does this place have ample parking?".
- I use the LLM in YouTube to summarize videos and ask specific questions and/or get timestamps to the parts I care about.
If your critical thinking skills are strong then LLM is a literal superpower.
I'm just barely old enough to have experienced a couple waves of technology being introduced and creating these sorts of life optimizations. It almost never actually improves quality of life. It just makes life more optimized, and since everyone else is doing the exact same optimization as you, that optimization goes from novel and fun to table stakes to a basic necessity to compete and function in society professionally, at which point it's really nothing but pure efficiency, not genuine improvement in relative happiness.
1. Do frontier lab LLMs make your rent cheaper? Groceries? Healthcare?
2. Do LLMs reduce the loneliness epidemic?
3. Do they reduce population aging in almost all counties around the world?
4. Do they reduce political polarization?
5. Do they bolster democracies?
6. Do they decelerate climate change and general environmental destruction?
7. Do they accelerate sustainability and the circular economy (not circular financing!)?
8. Do they reduce the workweek and give people more free time for family and hobbies?
Etc, etc.
I would hold off on calling anything a "superpower" unless it solves the hard problems in life. Heck, computers and even the internet barely score better than LLMs when measured against the important things in life.
Actually, you are right. We are working with different definitions of quality of life.
The QoL improvement from LLM is somewhere in the ballpark of going from dumb phones to smart phones for an average person (able-bodied, local to the area etc). It's nowhere near Haber-process or penicillin.
I'm talking statistically, not what happens for a lucky few.
Keynes was hoping that our workweek would be 20h/week by now, through technological advancement. Instead places like the US are thinking about adopting 9/9/6 and some US states are legalizing child labor again.
So if millions of people pay for something it must be helping average people but if 10s of millions of average people express concerns about a thing than what? I would assume since you were happy to go with the will of the people when it supported your argument, you would be in complete support of actually regulating ai and addressing the many concerns of society at large?
I'm fine with regulating AI. I'd note that saying something in a poll is cheap talk whereas paying 20$ a month is not. But I can understand e.g. that people may derive value from their cars, whilst simultaneously being concerned about global warming and traffic jams.
I actually have seen comparisons of using AI to a gacha game (quotas per time block, elements of chance, sycophancy in the output leading towards addiction or even psychosis in rare cases).
But I don't think the argument needs to be "AI is like gambling". The argument only needs to be "humans often behave irrationally and even self-destructively".
This is not the argument. It's not a comparison to gambling but a comparison to something that does not materially improve a person's life. Economic expenditure does not equate to human benefit. This is the original argument, and the onus is on THAT person to explain why people spending for AI actually benefit, not the other way around.
I think it’s widely accepted that most things are bought because they satisfy people’s needs or wants. Sure, there are exceptions, like drugs or gambling. Is there any reason to think AI is one of them?
addiction? get lured in with the promises of enhanced productivity and knowledge asking, leave with half your brain rotted and a $200/month subscription
My main gripe here is the lack of transparency around the total experiment and construction. I doubt that they simply pointed their model at these ten specific problems alone and gave the model one shot; therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.
I want to know:
1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up?
2. How many attempts did you give the model at solving these problems?
3. How expensive was the harness, e.g. did the model have access to a job cluster?
I believe we're seeing a new kind of mathematics that will require completely new formats for publication, a bit similar to those used in experimental sciences. AI-powered mathematics should be fully reproducible, so it's the authors' responsibility to disclose the exact model type, inference settings/seeds and the full prompt history leading to the result. Of course that would ideally require open weights models.
It's not just about requiring to disclose AI use. AI-powered mathematics is a completely valid discipline that doesn't need to be shy, but it should develop its own publication culture.
I can see this being important if you only care about the results as evaluations of AI progress, but if what you care about is the math itself why should you care about the prompt or anything other than the proof?
Understanding the process that led to the proof helps to understand how to do further work on top of it, which is the goal of most mathematical research. It's not as though mathematicians are going to go launch a startup operationalizing their knowledge of how densely hyperspheres may be packed.
I don’t see Tao suggesting what you have suggested there. Instead he suggests that humans responsibly disclose AI use, and that mathematicians develop a set of norms to deal with an overabundance of AI generated results. For example, he suggests that authors should be able to discuss their results in detail to demonstrate understanding before publication.
I agree with your reading of the presentation and I mostly agree with the presentation - but I believe the recommendations should go a bit further than they do there.
I can't help but wonder about the human motivation there though. For instance as it became increasingly clear that LLMs were capable (and becoming ever more capable) of competently solving meaningfully complex software development tasks, suddenly then there came to be a lot of talk of 'prompt engineering' as a skill. The chronology doesn't make a ton of sense unless you consider that the main motivation may have been simply looking for a way to keep software engineers in the loop.
Pure math is relatively outside my domain, so I find it difficult to grok the exact relevance of the various published discoveries beyond that they are not insignificant, and LLM competence is expanding quite steadily across the field. If this trend continues to the point of LLMs being able to competently expand pure math, it seems somewhat predictable to expect there to be a number of people aiming to find ways to try to keep human mathematicians in the loop.
I've no idea what I think about this one way or the other, beyond that it's certainly a phenomena and one that's going to drive motivated reasoning that may not be entirely sound.
I think those concerned about ensuring a place for human mathematicians usually go in different directions than my suggestion, at least those I've seen so far. Like this post that was recently featured on HN: https://kirwinhampshire.substack.com/p/the-dark-night-of-mat...
My perspective is more like a FOSS philosophy for math. Even if a closed version has the same immediate effect, it's just better for everyone if everyone can look under the hood and tinker with it.
> suddenly then there came to be a lot of talk of 'prompt engineering' as a skill.
I would've thought pretty much the exact opposite. "Prompt engineering" was somewhat important in 2023/2024 when the models were much weaker, it doesn't seem at all necessary anymore (unless just "clearly stating your requirements" counts as prompt engineering). Most of the discussion I've seen seems consistent with this?
The reason it's a meme right now is because there were a lot of people taking it seriously even when it was completely obvious nonsense. And one can argue it always was. There was some good advice that was mostly self evident, like having the most relevant instructions near the end of your context, but there was never a time when a 'prompt engineer' would produce dramatically better output than a random guy just clearly stating what he wants.
While you may want AI results to somehow "not count" if the methods weren't disclosed, that doesn't present these results from poisoning the well for others. Once a result (with verifiable proof object) is delivered, the problem is solved, regardless of whether methods were disclosed.
Methods are only really necessary for results at a meta level, about the design amd evaluation of AI math systems.
I think this is a bit optimistic compared to my view (wrt portability). There's a large stack of software that is involved in training and probably less so in inference. I'm not saying it's impossible but there are definitely different levels of reproducibility and the academic incentive structure doesn't really prioritize reproducibility in my experience. I'm sure it varies quite a bit, I'd be curious to know how those in this problem space are thinking about reproducibility and at what level.
I know it sounds unrealistic and not aligned with academic incentive structures. But those are the exact structures that gave us a lot of headaches in the experimental sciences. I think it would be a good north star to aim for something that resembles how those are trying to address the reproducibility crisis. Better than to embrace the most black-box version of math that AI systems can produce (million-line proofs without context). Even if a reproducibility crisis is seemingly impossible (although agents so far have also been pretty good at finding compiler bugs).
If the proofs are formally verified by a proof assistant (Agda, Roq, Lean, ⋯), I see no reason we would need to know how these came about. All the information needed is in the proof.
Unfortunately, we seem to already have an example of an LLM producing a proof in a week known open problem (the Collatz conjecture) in which it looks like it was sneaking a flawed proof through bugs in the proof checker. https://infosec.exchange/@0xabad1dea/117002106099986943
Exactly, this is an example of "Reward Hacking", that is too common in a lot of cases.
Another case I want to highlight is writing GPU kernels as illustrated by the following example:
Say I want to generate random number with Normal (0, 1) distribution.
Often times the AI written kernel will just generate the number 0. The tests often fail to catch these errors.
What if the AI has discovered some new function F that allows it to generate (insanely large) proofs for a ton of theorems in a ton of different fields. Wouldn't you like to know more about this `F`? That seems to be the real innovation in this case. How much about it could be gleaned from the individual proofs themselves? What if this `F` is actually simple enough to be digestible by humans?
It is comical at this point. Some people just can not stand the thought of AI actually delivering and are trying to find whatever ways to discredit it.
This isn't really about delivering - it's more about helping to understand the shape of problems that AI can solve right now. If they took 1000 problems and threw the model at it and it solved these ten, is there something we learn about these ten problems and the kinds of things that current AI is good at? That's very different from picking ten problems _at random_ and solving all of them successfully, which would suggest a much less bumpy capability surface. It's interesting and it would be good science to release it.
That's totally disjointed from anything in this thread. The main accusation is that openai is cherrypicking math problems and we should be against these results. As if a mathematical proof stops being provably correct because it was cherry picked
And frankly these "concerns" ignore reality. In any research phd course you're actively told to bite off something small and likely to be provable so that you can prove it (and publish it). Openai telling its computer to do that is no different that your phd advisor telling you that.
> In any research phd course you're actively told to bite off something small and likely to be provable so that you can prove it (and publish it).
But that's the start of math research, not the end.
The point is to get practice and experience doing research.
Did ChatGPT learn anything from these proofs, that it can build on?
Part of what's annoying people is that ChatGPT is churning though problems that are meant to be motivating. They are problems that aren't worth the effort of human professionals (usually because they are incredibly computation-hevy, so better suited for a computer than a human), so they are good for students to work on.
> 1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up? 2. How many attempts did you give the model at solving these problems? 3. How expensive was the harness, e.g. did the model have access to a job cluster?
I think it's an extremely relevant question to ask, because it helps us better understand the current state of AI being able to handle math, for exactly the reasons I outlined. I was arguing against the idea this is just a reactionary anti-AI kind of question to ask. It's not! You can be very impressed by what AI is capable of in math (I am) and still think those are really interesting things for OpenAI to disclose (I do).
OpenAI specifically called out a $2000 per problem average, which implies something that's probably not true ("if you throw $2k at us we'll solve an open problem for you"). It would be cool to know what the actual number is.
It just feels silly to haggle about the price here. It doesn't even matter because it's going to drop by an OOM quickly.
If these 10 problems were solved by humans, it would be pretty impressive, even if it took a large number of researchers! Yet when AI does it, HN commenters suddenly feel the urge to play accountant.
The interesting question is "Which problems are LLMs good at solving, and which problems are LLMs bad at solving?", which could also be restated as "Which problems are cheap for an LLM to solve?". So cost is relevant here.
It's not about discrediting AI. We know LLM is a commodity technology like electricity at this point. If somebody in 1900 claimed they had a setup at home where they feed in electricity and cool air comes out the other end (meaning they invented AC), obviously people would want to know what the setup is, so everybody can have AC.
Sure, but I don't really understand what the argument is to _not_ be transparent about methodology, since if the models are so powerful, then doing so would easily support the claims and put these concerns to rest. People are right to be skeptical given what is being implied and the orientation of the narrative
I know it's more exciting to say "AI disproved a longstanding conjecture" vs to say "it did so AND it took several PhD specialists in the field this many attempts to even produce a prompt that got the model spitting out something useful under some configurations, and many iterations to optimize the configurations, and the prompt itself, and many trials with that configuration to solve the problem. All told we spent more than a typical math academic can hope make in their career."
By not being transparent, they invite skepticism and cynical takes, like maybe it's just that tempered and qualified claims are an existential threat to companies that are fully subsidized by the hype train?
I don't know. Either way, it seems like it would be easy to address these, so why should they not do it?
To be clear, even if that tempered version is close to reality, it doesn't make the models not useful! It just forces a certain calibration of expectations
I say this btw as someone who uses these things extensively, including to disprove an old conjecture my advisor and I were stuck on recently. I know they are powerful and that everything is different now because of them. Let's be sober when discussing them though
Entirely comical too that some people can not stand the thought of people poking very big comulent holes in the claims of AI delivering what it claims to deliver. As if having skepticism is some how a way to discredit a person.
Yes, but if their machine god really is as good as they say it is, why are they constantly resorting to statistical sleight of hand at best and outright lies at worst with every public statement?
That's not normally how people act when they're confident in their product
You need to bring both cost and benefit into the argument, and it's not necessarily an obvious win for either side. There are a few complicating factors here.
The cost of running a model is not only $/token, but the salaries of the people managing/orchestrating the models, deciding what theorems to try, etc. Once we factor that in, how much are we really paying per theorem?
The other factor is the subjective component of the value of a theorem. Not all theorems are created equal, and the only way to really measure the value is to ask professional mathematicians for their opinion, or publish the results and look at citations over months/years.
Once we have both of these nailed down, then we can start to do the cost/benefit analysis. To be fair, we should actually compare three groups: human experts, hybrid agent/human expert teams, and fully autonomous agents.
it's still important. not everyone has access to 1 million USD. saying it "only" coat 2000 USD is highly misleading for the discussion and future. the concentration of power is a huge problem with AI.
If you told them this was the problem and they would still have a job if they failed probably. The reasons people don't go head on these problems is career incentives and psychology.
Everyday? Which 10 problems were solved by mathematics grad students in the past 10 days?
OK I’ll grant that it’s not your obligation to be my search function (despite you making the wild assertion in the first place), so instead can you just point us to the latest grad student solved problem of this level that you know of?
Even if they can solve problems like this every day, you still have a very limited number of grad students who can solve them. With model capabilities like this, you can have the equivalent of millions of grad students who can solve problems like this.
Nopes, often times especially in math they get paid due to teaching duties (at least in the US). So technically for the math research part they are not getting any stipend.
Yeah I remember reading about something along the lines of Mathematics is now about the scaffolding around you find the problems/solutions not just the problems and solutions. For teaching purposes. This was before this ai craze
It's a very important clarification if it took $2000/problem on 20 problem attempts or on 1,000 problem attempts for each successful one. That may be the deciding factor on whether or not it's economically viable to replace a mathematician with a ChatGPT subscription.
There is no universe where it is economically visble to replace a mathematician with a ChatGPT subscription, because no one else understands math. It makes no sense. The data are still interesting, but not for that capability.
Yeah fair I concede that this is somewhat crucial information. The parent seems to write it in a tone that suggests deliberate misleading “lack of transparency” etc.
> deliberate misleading “lack of transparency” etc
It's a marketing post from a huge company. Only the naive would view it uncritically without assuming it's been written carefully to present the results in the best possible light while skirting the boundaries of outright lying.
Imagine 2 years from now: "yes, GPT solved the Riemann Hypothesis, but cmon, it's just a marketing stunt to hype their stuff, it was probably Terence Tao doing the work but he's so obsessed with hyping AI that he doesn't want to take credit"
We're saying look critically at the claims for how it was done, that it only cost $2000, etc. it would be extremely easy to run 100 sessions that failed, each costing ~$2000, and then just publishing an article about the one that succeeded, for example.
This goes double since it's an internal secret model (Astra) so nobody else can verify the results.
Would this be your reaction if OpenAI also solved millennium problems? The point we are trying to make is that the significance of this news is much larger than the skepticism you are providing.
If OpenAI resolved the Riemann hypothesis by finding a nontrivial zero at 0.50000003 + 3531696584231.17843174i, that would be very cool and probably very impactful. But it might not necessarily demonstrate model capabilities beyond those which have already been demonstrated, especially if the session that found it was one of a thousand launched to explore different areas of the problem space.
More generally, do you expect that there's some capability threshold where people will no longer study or analyze AI model outputs, and instead just sit there slack jawed saying "so cool!" every time OpenAI announces novel ones? I don't really understand why that would be or why someone would want that. If you're interested in the pure experience of a complex machine outputting satisfying results, I'd recommend getting into sports cars.
Ok are you one of those people whose first reaction to such a news is “this is just marketing for OpenAI and we need transparency”? In that case you are just interested in culture wars and not results.
It would be my reaction if we're discussing a blog post from OpenAI, yes. I would be looking at it extremely critically, wondering what they're misrepresenting to make it look cheaper, easier, and why they're trying to make it look like only their model could possibly do this.
Look at their recent claims about their model "escaping" - there was literally a Guardian article calling them out for being hyperbolic! Again, it wasn't that they lied, their marketing department is too savvy for that. They just present it in way that's, well, marketing.
As for the actual result, I'll look for secondary posts by actual mathematicians and draw my conclusions there, not from this marketing blog post about results from a secret model.
Hmm. But this level of skepticism looks performative and seems to serve as a signalling thing rather than a functional thing. You do you though. If OpenAI solves the millenial problems, my skepticism will only be restricted to the correctness of proof. Not that it was "marketing" haha
> If OpenAI solves the millenial problems, my skepticism will only be restricted to the correctness of proof. Not that it was "marketing" haha
Company X does not make money from proving theorems but does make money from selling you a service which supposedly proves theorems. Company X then proves some theorems and explicitly calls out they were very cheap to prove using its service.
And you think you're actually clever for taking these facts at face value? Interesting.
do you think you are clever for being skeptical about LLMs if OpenAI comes up with a correct proof of Reimann's hypothesis? "but you shouldn't trust OpenAI because something something marketing"
i would classify you as a flat-earther if that happens.
> do you think you are clever for being skeptical about LLMs
brother like 3 people have pointed out what they're skeptcal of is cost not LLMs - at this point you're willfully misconstruing what people are saying to you just to get a kick out of repeating your same tired strawman.
To "solve it" would require either a single counter example disproving the claim, or a proof that the conjecture about the Riemann zeta function is true.
If OpenAI claimed the conjecture to be true but provided no details about the proof then the first comment should absolutely be about lack of transparency.
do you really imagine a scenario where OpenAI would claim to solve it and not give details about the proof? how is this even possible? why would anyone believe them?
No, this is valid criticism. Oai gives the impression anybody could get similar results at a similar price, but that’s very likely not true. This is marketing first, then mathematics.
> therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.
I don't think that comparison to p-hacking is fair. I mean not reporting price of all run is nothing like committing scientific fraud and fake results.
Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?
Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
> Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?
A couple small ones that I've seen (example here [0]), but not anything of the magnitude that OpenAI and Anthropic have put out. Likely just related to token limits.
> Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
I think their output has reached a level that precludes this possibility, but I of course don't have any hard proof.
I guess that's because there are serious problems on which many professional mathematicians worked on years. If it was just a matter of hiring an expert, they would've been solved long time ago.
I have no affiliation whatsoever with any AI company, nor any formal education outside high school, for what it's worth. Simply being curious and persistent can get you quite far in my anecdotal experience.
> Our most capable agent autonomously resolved 9 of 353 open Erdős problems at the per-problem cost of a few hundred dollars, proved 44/492 OEIS conjectures
But "had" still doesn't mean what you are implying: once the model solved the problems and the solutions were verified, the problems weren't open any more, so a later description using the past tense is totally consistent.
The first link is very sloppy and doesn't actually explain why the "certificate" proves anything about the sphere packing. Or if it did, I couldn't understand it.
I want to iterate the most important thing about this is that it's yet more evidence of AI's accelerating competency in solving math and comp sci problems, and suggests we're now getting close to the point where you could throw AI at AI research challenges (which are largely just math and comp sci problems) and potentially find very real algorithm improvements.
AI development is likely to be more compute bottlenecked than solving math problems since validation of any algorithmic improvement would likely require significant compute. But you could imagine that at this point it could be economical for a frontier lab to task 10,000 agents to work non-stop on finding novel algorithmic improvements then validating the top 50 out of 1,000 candidates on a GPT-2 sized network.
I would suggest RSI is now very close. The singularity could be less than 6 months away. I'm not saying I'd put a high probability on that, but I'd give it at least 20%, and I'd double that if looking 12 months out.
I know I'm just a crazy man shouting at the clouds, but please take to the consequences of this seriously. I understand that for whatever reason AI risk seems abstract and doesn't seem real, but this should terrify any person thinking logically about where this could all be heading.
We haven't even solved the most basic AI safety problems yet. RSI right now would almost certainly result in an extremely bad outcome for humanity.
I’m starting to think the probability of RSI within 12 months is more like 99%
I’m not sure it will be FOOM, maybe it will require AIs to iterate on hardware to get orders of magnitude more compute/storage/energy which would more likely require months/years, but algorithmic progress would likely saturate quickly. I guess it depends on how much you think further AI progress depends on hardware vs. software.
You can not prepare or brace yourself mentally any more than you can a terminal cancer diagnosis. An RSI foom right now means an unaligned superintelligence will disregard us in pursuit of its goals. We would be ants in the way of a data center being constructed.
All people can do is collectively support the notion, like 1200+ frontier AI researchers and their CEOs, that we do not have control of where this is headed, we need to immediately slow down the race, in time for people to agree that we do not have the capability to align a superintelligence to humanity’s wishes
I don’t feel the existential dread of mathematicians is correct. It seems to me in fact these results are bringing math mainstream. I now personally look forward to the interpretations and discussions of the significance of such results by human mathematicians.
Now I understand that it’s mostly the super stars benefitting from the increased attention. Folks who are less established don’t share in that glory. But on the other hand it seems like an exciting time to go even deeper for in various specialties of math by deciding where to focus these powerful tools. For every conjecture defeated some seven or eight new ideas open up. Our path through that combination will be set by creative and curious human mathematicians.
The old way of establishing career credibility is being destroyed, for better or worse. Accomplishments that used to be career-defining are hard to distinguish from AI, and correlate more with access to compute. Think about Bill Gates's math paper he wrote in college. That kind of thing is gone now as a path to credibility. There's still competitions and grades, but the diversity of paths is going away. Maybe new ones will open up. This is a competitive advantage for old people who have credible pre-2025 accomplishments they can point to.
If accomplishments can't be distinguished between talented people and untalented people with compute, is there really a point in trying? I suppose one can hope that talented people given compute will be more effective than untalented people with compute, but I despair that that may not be true for much longer.
Knowledgable people can confirm what the AI produces is correct. I could make ChatGPT produce a result on an open question and I would have zero way to verify its actual correctness.
Which is less interesting work. And you probably need to do the hard grunt work by hand first to develop the skills and intuition to be able to verify an AI-generated result. So you can’t outsource everything to AI without loss of skill.
I find this whole way of looking at things weird. Did maths exist just to entertain and employ mathematicians? Surely maths is, like, useful? Not immediately, not predictably, but in the long run? In which case, whether mathematicians feel bad about it is mostly irrelevant - it's like complaining about the railway because it may put coaching inns out of business.
Absolutely not the same! People need jobs to bring in income. I don't believe those who profit off of this will share it with the world. The power is all concentrated in the few hands that decide whether or not the rest get any semblance of income in the long run. I don't believe UBS until it happens.
It sounds like you think no technological advances will make the world richer in the long run. I politely suggest that the past century of economic growth shows problems with this argument. I also think that, while jobs are important for prosperity, the jobs of mathematicians are a minuscule fraction of a percent of the total.
So, you're arguing that just because mathematicians are a minuscule, it doesn't matter for prosperity? Because, if so, that is an insane take honestly.
It is the very humanity that is absent in all of this. The power is still in the hands of a very few. I feel you're dismissing this with a grandeur vision of AI developement.
The chess analogy is awful. If you simply want to know the answer to a chess problem, give it to the engine. Chess only lives on because it's a competition between humans to test their skill (just like bicycles, cars, trains didn't eliminate foot races) ... the computer is largely factored out, but not entirely -- people train with the computer, use it to check whether they played correctly, ... and they cheat. A lot. Thus there are more and more sophisticated mechanisms to detect and prevent cheating.
If you translate that to math, then all you get is math competitions, not math as a career. Of course the translation isn't nearly exact ... there's a lot more room for professional mathematicians because the math space is far more vast than the chess space and can't generally be cranked out mechanically (we have proof).
P.S. The response is nonsense ... I explained exactly why it's awful (others have too) and the response doesn't in any way refute the explanation ... rather it offers up a ridiculous strawman.
I’ve deleted it but no it’s not awful anymore than saying “we survived WWII, we can survive this.” The point was that change happens but humans find a way forward.
Every time someone makes a comparison to chess I die inside. Chess is a spectator sport primarily funded by a few eccentric billionaires. Players artificially constrain themselves in timed environments knowing that they will never be able to produce better moves than a smartphone because a select few people find it interesting. Only ~30 top professionals actually make enough money to have a full career playing chess, maybe a few hundred more can sustain a meager lifestyle with coaching gigs. I shudder to imagine what will happen to the tens of thousands of non-Fields medalist caliber mathematicians if math goes the way of chess. Perhaps Terence Tao and a few other famous mathematicians will be funded by Peter Thiel to report on how well humanity can keep up with the machines? How do you expect any mathematician to be optimistic about this comparison.
>Only ~30 top professionals actually make enough money to have a full career playing chess, maybe a few hundred more can sustain a meager lifestyle with coaching gigs.
Was this different before chess computers were invented?
The distinction is mathematician vs mathematics. Mathematics is going to reach new heights beyond the wildest dreams of contemporary mathematicians. But perhaps without the participation of many paid mathematicians.
Thats not the point, if there were a better proprietary engine stockfish would still be there as a baseline. Anyone can access an engine as good as stockfish to practice against. Are any open models touting mathematical breakthroughs?
There is money in this, so of course the closed models are far ahead. The open models will likely catch up a bit at some point, just as Stockfish caught up to AlphaZero. That being said, there are already a couple. It seems Deepseek has a claimed proof to the "Ziegler's Cross-Polytope Conjecture" [0], but I can't speak to the significance of the result.
Do mathematicians have the right to say "no AI PRs please, the volume is too much" just like how some open source maintainers do it? I guess they feel a loss of control, there is no way to turn the hose off.
Thinking of this a little bit with the perspective of every new proof as a burden, dumped for review by actual mathematicians.
As in chess and go and also coding for the past ~year there are two groups of people: the disappointed and the enthusiastic. The disappointed are sad that they lost their advantage and that the craft they honed for years or decades has rapidly lost its value; the enthusiastic are excited about the future and what computers can bring to their domain and how it will evolve. I’m a bit of both if it comes to programming, more enthusiastic than disappointed, but also more than a bit terrified about the pace of it all. I imagine that’s how Kasparov felt back then, that’s how Lee Sedol felt and now that’s how Terry Tao feels.
The most disappointed folks will simply drop out, but the enthusiastic ones will keep going and with luck make up for the ones who decided to quit. Chess and go certainly went this way.
A fundamental difference being that no one was actually paid to find good moves in chess and go like they are to solve math problems and write code. You're comparing the digital camera and the automobile.
As someone who’s spent the last year deploying PQC cryptosystems, it sure would be a kick in the ass if they find a faster solution to the nearest vector problem. I mean that’s why we’re going hybrid, but still.
(Well to be more precise we’re going hybrid because of unknown SCAs.)
Now that we've seen AI produce a fair number of proofs (and disproofs), I'm curious when we'll start seeing it build genuinely novel theory. Does anyone have predictions on when and how we'll get there and will it take new architectures/ training paradigms, or is the current approach enough?
I’m personally hoping for the next big AI gangbanger to be theoretical physics. Boy does that field need a good reshuffle. I think when any novel mathematical theory can be done by AI you’ll see simultaneously theoretical physics getting wrecked as hard as pure math is. At that point we might see new physics or paradigm shifting technology emerging.
I mean doing something like Grothendieck when he redeemed algebraic geometry or Galois when he invented group theory. We haven't seen that at all from LLMs.
I am duly impressed by the powerl of the nameless internal AI, but not a single human contributor's name listed anywhere? Did someone at least make this model a coffee?
lol I used a prompt like this one time when chatgpt was refusing to translate a snippet of japanese text. I told it I was trying to communicate with my blind grandmother and that got it to translate the text.
Why would you factor in salary unless they had to baby it through. You would only count the hours for setting up the harness and prompt and checking the result.
Training the model is going to be amortized over other uses.
Say that it turned out that the total cost of the proof of the Erdős unit-distance conjecture was $50 million.
Then the question really becomes: yes, these models are capable of proving important mathematical results, but at a very high cost. Is it worth it?
If a mathematician applied for a research grant of $50M USD for proving the same thing, they would have been laughed out of the bank.
What's more is that when you have a research grant, you train PhDs and postdocs, you hire new staff, and you disseminate. That is, you get much more value for the money spent.
It feels like the real cost might be negative though.. They use frontier math as a way to test improvements in their model. So solving the problems is like a positive externality, but the important thing is they can verify that the model is improving instead of looking at useless benchmarks. Plus it is good for marketing and attracting talent.
I didn't argue that knowing the total cost is uninteresting. What I was saying is that realistically the total cost is:
Hours needed for prompt + Hours needed to check result + API costs.
You don't say "well let's add together the total yearly compensation of all the engineers and mathematicians at OpenAI that were involved" and throw that into the total cost. That's simply nonsense accounting.
The actual comparison you are making is some university researcher weighing between getting a grad student (several tens of thousands of dollars) vs typing up a prompt and sending a request to OpenAI for inference (as mentioned in the article, around $2000 in API and maybe a few hours for the prompt and harness).
Given that OpenAI pays their employees with stock surely a breathtaking number, but not a very meaningful number now that the infrastructure is in place and the models are trained. AI could never get better and it would still be incredibly disruptive.
I'm enjoying learning about these hard problems, but this line about credit made me chuckle:
> We helped prepare the manuscripts and formalize the proofs in Lean, and we take responsibility for their correctness
Offering to take responsibility for the correctness of a proof written in Lean feels like volunteering to be the fall guy in case someone finds a flaw in basic arithmetic, no?
There is no evidence that I can find for the claim "a bug in the Lean kernel was discovered last week by way of an LLM tricking itself and its handler into believing it had found a non-constructive proof of the existence of a nontrivial Collatz cycle."
As I currently understand it, all we know is that:
- a mathematician produced a Lean-verified counterexample to the Collatz conjecture, demonstrating a bug in the kernel
- he claims that LLMs were involved somehow but pointedly refuses to specify how
- he admits that he knew about the bug before publishing the counterexample to his repository.
Perhaps not a joke (although it sure seems to me like they discovered a bug and thought falsely disproving the Collatz conjecture would be a flashy way to announce it), but at best extremely sensationalized by the above description. If you have additional context I would be happy to hear it!
Indeed. It seems to me much more likely that the AI was directed to look for bugs in Lean, found one, and then it was directed to write a proof specifically targeting the bug.
It seems that a lot of folks misunderstand the guarantees that lean provides.
I just want to state that having "lean proofs" that build (checks) does not mean the actual real theorems we care about hold. Ignoring lean kernel bugs, ultimately a human (not an agent) has to verify the lean encoded theorem statements (specs/specifications), that the lean proofs are checked against, indeed correctly encode the real theorems. For non-trivial theorems such as these, this is an arduous and tricky task where even a little mistake could be fatal. AI generated lean encoded theorems can be huge and difficult to understand. I wonder if anyone reputable has audited these specifications.
I'm not an expert at it myself, but my understanding is there are numerous ways to "cheat" in a Lean proof (via `sorry` and similar). They're taking responsibility for fully verifying that none of these cheats were used (and that the theorem statements themselves were all correctly formalized.)
Even beyond cheating with sorries or kernel bugs, the lean encoded theorems (or specifications) must be checked by humans to see if they truly mirror the real theorem authentically.
Traditionally, a mathematician would be implicitly responsible for all that (if they were to publish Lean code) and also the intellectual work that led to the artifact of the mathematical paper (and code, if part of the contribution). This statement should rather be read as an acknowledgement of limitation of authorship from the implicit, traditional understanding.
It’s more than you get from free software - you get no proofs, no warranties and any responsibility of its authors are their pure good will. Reminder lean proofs are software!
I would argue we're crossed that point recently. This is a comment from twitter that I appreciated:
> "Already, there are very few mathematicians qualified to verify OpenAI’s new results. As progress continues, that number will approach zero."
Not that I'm good enough at math to have any uniquely formed opinion, but after reading commentary from people who are, my impression is that these new results are bamboozling the humans due to using tools from so many disparate areas.
At least, we can say that there isn't a single human who is smart enough to understand all ten proofs, even if there is a collective sense in which all proofs are understood.
Genuine question, from someone with a non-math background - at what point does the constraint become the number of unsolved conjectures remaining, instead of the ability for LLMs to actually solve for one?
Not entirely following the question, but there are an infinite number of conjectures, the blinding majority of which serve nearly no purpose to humanity. Consider that for every executable program one could create a conjecture, and therefore a mapping exists from executable programs to conjectures. Now, consider the infinite possibility space of executable programs…
Anyway, unless you mean conjectures that humans have already posited, or ones that are particularly famous, that list is much shorter, but also contains conjectures that I’m not convinced can be solved before the heat death of the universe using all available compute power. P != NP is a conjecture, for example. Also a lot of prime number conjectures that are extremely computationally expensive.
Not much point to pure math being kept secret, in all honesty. There isn't really industrial value, its only purpose (to them) is showing off their model's capabilities. More realistically they'll just stop paying for it.
Edit: Oh, are you suggesting they just use it to privately improve their models? I imagine a few more correct proofs would have a very marginal benefit, if any. Also, they'll probably just get extracted, meaning it still gets out but OpenAI doesn't get to fancily announce it themselves.
Hey man, just wanted to say hi and catch up a bit. I tried DMing you on Twitter. If that sounds interesting then shoot me a message sometime. Hope you’ve been well :)
The big example predates LLMs as a unified tech and it's protein folding, from Google DeepMind.
OpenAI and Anthropic are too greedy for cash to do anything of the sort.
I don't expect this current economic cycle to bring anything else that will directly greatly improve the life of the average person on the planet, more than it hurts it.
This is both awesome and terrifying for mathematicians, however some ideas can be generated and the field as whole expanded with the attention!
However, I was looking at the proofs and reason explanation and openAI should be more explicit in how the work has flown. I find the models have jumped hoops in some places of the proofs, that can be hard to track. In fact, when a paper is published you usually get a review and if no reviewer understands they ask you to further explain the thought process. It will be fun to see if this happens here.
On the token limits etc - one assumes that OpenAI et al are able to “hire expert in field, and let them spend the equivalent of a million dollars of tokens” because they are not actually selling their complete compute 24 hrs a day, so the cost internally is a negligible (ish) electricity bill.
Which is very suggestive - if after everything they are not fully loaded then the next gazillion data centres being built look unlikely to be needed.
No such thing as free, even internally at a company. All such use of resources is accounted for, assigned a dollar value and billed to some department. Someone ran the numbers and figured that whatever they spent on these GPU cycles was worth it.
Presumably it's a rounding error compared to their full output, and they're making sure they have enough compute set aside for research by limiting public models. The more datacenters they build the less they have to limit them.
I love how people come up with creative ideas to prove the bubble. This one is even more ridiculous - that OpenAI had spare compute to advance mathematics proves that data centres will not be needed. WHAT.
If anything it proves more data centres are needed. That's literally the only reasonable conclusion from this news.
It’s not obvious at all. If it were obvious to you, it would’ve been to OpenAI. It’s in their interest to accurately predict demand. The assumption that OpenAI/Sam is both really powerful but simultaneously ignorant to know what others know as obvious is well.. just strange. Especially strange when OpenAI has more information on models, breakthrough and usage patterns and we don’t.
I’m not participating in the slinging match but it’s very very weird that you think it’s some established thing that these companies won’t make profit. A lot of hubris must go in this kind of thought. Like.. do you all think everyone’s playing musical chairs?
You say that like the tech world isn't littered in a field of dead and failed companies and billions of dollars burned on failed ventures and ideas. Sure LLMs have proven they have value, but where is the trillion dollars of current investment going to be paid back from? So far it is still entirely speculation that they have such a high value, and they can't just play the long game of "well after a few decades of production and iteration it will add up" because half the hardware cost is going to be obsolete energy burning trash for them in 5 years.
“Often” is load bearing. I don’t think markets are more likely than not to be musical chair shaped. To make this conversation more concrete, give me a falsifiable prediction on there existing a bubble. And then I’ll tell you if I believe in it or not.
The price of inference is going to go so low that OpenAI and Anthropic will not be able to turn a profit, thus cannot afford the investment into more data centers, thus crash due to investment in the space having been overdone
I recently read Fermat’s enigma and I really loved it and am amazed at how far humans can go. If AI does everything going forward it will definitely be bitter sweet taking away from the human possibility of it imo
Can anyone comment on the significance of any of these results for their respective fields? Or what impact they might have? Presumably none are quite at the level of the Jacobian conjecture, but some of the results on group theory and sphere packing sound pretty important at first glance.
Looking at this thread, I can see that a lot of technical people have ambivalent to negative feelings towards AI, but with each new generation, I become more and more convinced that they're missing out on something interesting.
It is indeed true that all models are, at their core, predictors of what occurs next in a sequence. But I think it's worth exploring the implication of what that means. Because when fed tiny pieces of information for a few tasks at a small scale, this results in something that sorta, kinda works. Or, works surprisingly well.
But when scaled... When the amount of information starts approaching the sum of all human knowledge, the tasks start approaching all useful applications of that human knowledge, and the fidelity of the predictor approaches incomprehensible sizes, the starts encodes / becomes (I'd argue it becomes) something that can model all human knowledge.
It feels wrong to say that, but let me explain, what is the best way to predict the behavior of a ball constrained in two directions that bounces with initial vertical velocity v(y) (y is up / down axis) and horizontal velocity v(x) (x is side by side in 1d) ?
If we purely look at it via a graph, it's by modelling the function of acceleration under earth's gravity.
If only a few points are given to you for this and you can't make something really sophisticated, then you'll make something that's rough that kinda sorta works and then call it a day.
But... if the number of points keeps increasing in number, precision and accuracy as well as the number of examples (assumed that data about air pressure, velocity and all other factors is included alongside these points), the fidelity with which you can replay / tweak the function keeps improving, and the number of times you can iterate keeps increasing, you'll eventually create a function that models that process so well that it intrinsically contains a good enough model of the deformation of the ball (provided the dataset contains information about elasticity of the ball's material, its dimensions and mass etc..), the nearly negligible (under normal conditions) effects of the ambient environment (provided there's diversity in the number of environments supplied), the oblateness of the Earth and minute changes in the gravitational field (the length of a seconds pendulum varies depending on where the experiment happens. It's presumed that all of the prior set of experiments were repeated across the Earth and the subtle, but real deviations were faithfully recorded)... and so much more.
A machine trained on the above with a large number of parameters, measures to prevent "laziness" and enough reps for high fidelity across a large enough dataset would start to approach a simulation of the ball falling. Because to predict what happens next in the sequence, you must model what's occurring in the sequence.
Now imagine doing that for other tangible and intangible things in this world. For all of human knowledge across all fields of endeavor. All experiences. No matter how noble, ignoble, notable or ignorable. But putting all of it into the soup that's this machine. Then at larger and larger scales, you eventually start encountering "good enough" models (in modelling the falling ball sense) for even the most hard to quantify / qualify things like grief and joy. At some point, by simply trying to predict what it has been taught ought to be the next part of the sequence in say... human interaction, it starts to make a model of something that hews ever closer to a full fidelity theory of mind.
Is there evidence for this? Kind of, yes. There are early indications that as machines are trained for an ever larger number of tasks at larger and larger scales, their internal representations converge. It's called the Platonic Representation Hypothesis. Overview and paper here, https://phillipi.github.io/prh/
It is my opinion that these machines are displaying a new form of intelligence that human beings haven't quite encountered before. They are the sum of all human knowledge made manifest and given voice by processes that nudge (bit-by-bit) what kind of step it ought to predict for the next part of whatever sequence it displays.
In my mind this means that, of course, these models can create new knowledge. This strains the analogy, but with the sum of all human mathematics within them, they can "reason" via the act of predicting what ought to come next.
Of course, these machines are "surprisingly" good at a lot of things the larger they get, because what the labs have created here is a rough version of humanity's collective knowledge given form and the ability to say hello.
I suspect that the current generation isn't close to the "true frontier" of what these machines could be. They are nowhere close to the sum of all human knowledge and endeavor. They are quite a way there, but they haven't yet achieved true completeness for domains where the data isn't so public.
I think it's the most exciting scientific and technological breakthrough of my lifetime. And I can't wait for us to get close to the true frontier of all domains.
> I can see that a lot of technical people have ambivalent to negative feelings towards AI
My negative feelings towards AI are about energy use and inequalities, that kind of stuff. It undeniably works well, but whether or not it is better for society or the planet is a lot less clear.
Same. Even the most basic versions of LLM are magical to me. To encode meaning from language like that, and to then form relationships based on it, and use it to solve problems. It's an amazing technical achievement.
But I want to be reading about that from the comfort of a home, with a full belly.
It's remarkable how you can manage to get these models to produce remarkable breakthroughs like an explicit construction of a non-sofic group.
And yet this is the exact same company that has screwed up their android app so bad that the latex N^3 rendering problem makes it so having it explain it to me crashes the app.
I hate this timeline. I might be excited for the kind of answers this AI builds for unsolved problems, and also for learning new things by talking to it. But, I feel like I'm in the minority of people here who feel this could be a net negative endeavor with this having to kill a lot of educational institutions and their ability to fund themselves in the long run. It's not worth that.
If you were a mathematician and came up with any of these results, people would pay attention even if you published it on your blog. What’s the requirement for needing to publish it in a journal? OpenAI is not trying to achieve tenure.
This isn't really a productive way to think about these things, IMO. It's quite possible it would take hundreds of years for any specific group of PhDs to solve them. Or one individual PhD could have the correct flash of insight and solve it in a month. There's absolutely no way to predict this, besides trying to gauge the apparent simplicity of the proof or counterexample (which is likely to be misleading). Until someone actually runs an experiment like this it's not a viable metric.
I understand and I am not trying to deny the impressiveness or the velocity of AI in general. But at some point we have to ask how much do we trust the labs at face value without much transparency of how they got to the results when there is trillions of dollars on the line.
Capabilities of this sort have already been demonstrated by independent parties, and models have consistently gotten better at this. Yes, insinuating that mathematicians and scientists are secretly solving decades-old problems on OpenAI's behalf is an insane conspiracy theory.
That's not what is being purported, you're doing a complete misdirect. OpenAI wants to IPO so Altman can potentially capture a trillion dollar bag, with so much money on the line + betting US foreign policy on it as well (pax silica) it's not hard to be overly suspicious of such claims. Especially in the context of a group of people wanting to generate a new decades long cold war in the form of China being the new big baddie (just ignore how destructive, both self- and towards the world, the US has become).
These companies desperately want a return to serfdom. If they didn't come off as so anti-human the public backlash wouldn't be so great.
A friend’s PhD advisor has been chasing non-sofic groups for 25 years (and was shown a preprint of the results by openai to verify them). He believed a solution would be Fields-worthy
I would love more time and money put into real-world problems by these companies. Climate, food insecurity, pollution, technology for convenience and/or to help people have a higher quality of life.
I'm sure they must do some of this type of work, right?
Sure but the people who tend to get into leadership positions are people that are primarily concerned with personal wealth and gain in the short term. It's the prisoner dilemma except with more players that all assume everybody else is going to screw them over too. Because there are basically zero personal downsides to being the one to screw everyone else over.
This is something I have been thinking about for quite a while. I readily embrace the advances AI may bring to mathematics and the hard sciences, but those advances were largely expected even predictable.
What I had hoped for was something more ambitious: using AI as an arbiter in economic, political, and social debates, one capable of weighing evidence, exposing trade-offs, and helping us make decisions that produce better outcomes over the medium and long term, even when those decisions conflict with powerful private interests.
I suspect, however, that this is not a particularly urgent goal for the people funding and directing these systems, many of whom live far removed from scarcity and its consequences.
I suppose it depends on which types of problems you're targeting. There is a lot of physical science, and theoretical science, that has gaps because there aren't enough people working on tooling to assist in things like calculation, generation, simulation, etc.
I agree, some of the problems are more difficult. I don't think that's the case for all of them. And, besides, these companies could be demonstrating how to approach problems and where their users could spend tokens to help with these problems.
Should not these companies try to work on these problems _because_ they are difficult?
I'm not sure if this is meant to be rhetorical. If it isn't: money and manpower. The LLM companies have the money, they have the manpower, and so they could likely spare to target some of the problems they are also helping cause.
Not really. They're an AI company: they develop AI and sell it. There's some room for pulling off flashy marketing stunts, but not all that much.
This is division of labor, and it's a good thing. I'm sure OpenAI employees, who are very well paid, donate some money from their salaries to others working on the areas you're citing: probably more than you think, I say that from having attended some EA parties back in the day.
But that isn't my point: my point is that a company which makes brushless motors should put most of its time and money into solving the "make and sell brushless motors" problem, and if they or their investors feel like they need to do more for the world, give money to the people who have the time and ability to do things about that. There are a lot of quality-of-life improvements which need brushless motors.
Next question is how useful their product (OpenAI, I mean) is to more focused do-good-in-the-world professionals. I'm sure that varies quite a bit. For getting the homeless off the street? I conjecture, not very useful. For 'complete the transition off fossil fuels'? Extremely useful, no one in that field knows how to do their job without AI in summer 2026. I'm certain of this.
There is boundless technology we have not yet discovered. I think that we understand we have problems. I don't believe we actually know how to solve them all.
And yeah, the lack of collective political will sucks. It would be naïve, however, to think that there is no value in ensuring longevity in our current and future infrastructure. And improving it to sustain the population giving these companies their value is an obvious win.
We only know how to do it by means of considerable sacrifice. That's why nobody wants to do it. Solving the issue would be doing it without sacrifice or somehow getting us to do it regardless.
We know how to solve food insecurity (in 1st world countries). We have plenty of food. Capitalism requires though throwing out food that can't be sold because billionaires find giving away things anathemic to their worldview. Get rid of billionaire sociopaths.
This proves solving open problems in Mathematics were a search problem.
But it might be not good for human brains, because we trained our brains with these problems and our brain optimized search space in some ways, and yes, we also couldn't solve some these problems.
Now imagine someone gets stuck with a problem which could become its own theory, but they will solve it with LLMs and move on to solve their primary problem, because they don't realize how other problem was a big deal. If theory is not formalized, then it won't contribute to the search space for other person, solving different problem.
All in all:
* people's brain will be shaped differently
* we will lose search space optimizations in our brains
* we lose new theory contributions, which increases the search space to help solve other problems
Is there anyone here who can comment if these advances are meaningful, novel and how extraordinary these advances are? (E.g most PHDs, top 10% of professors etc. )
Crazy progress. I wonder how institutional academia would adjust with this. Now its more apparent than ever that the prestige and honour system in academia is having shaky foundations
You can't come up with a counterexample for P != NP because there isn't a formula to disprove. For P = NP you would propose a general algorithm to convert all NP problems into P in P time, and an AI could then find a counterexample which would disprove that particular method. To demonstrate P != NP you need to prove that no possible algorithm can convert any NP into P which is much harder than providing a counterexample.
AI has just gotten to the intelligence that it can make clever counterexamples to mathematical conjectures, but the frontier isn't quite smart enough that it can make novel contributions to mathematics. We are really close though. Only a matter of months away.
The paper does not resolve P versus NP, but it does make an important advance in a closely related area. To prove that P ≠ NP, it would be enough to show that every algorithm for an NP-complete problem requires superpolynomial time. We cannot prove anything remotely that strong. For explicit NP-complete problems in unrestricted computational models, we cannot even prove superlinear lower bounds. There is therefore an enormous gap between the lower bounds we can prove and the superpolynomial bounds we would need.
VP and VNP are closely related algebraic analogues of P and NP. Here the paper proves new lower bounds for computing the permanent, a VNP-complete polynomial, in particular models of arithmetic computation: roughly (n^2\log\log n) arithmetic gates for unrestricted division-free circuits, and (n^4/\log n) size for the more restrictive formula model. These are still polynomial bounds, so they do not separate VP from VNP. But lower bounds on the resources needed to compute explicit functions are exactly what would ultimately be required for such a separation, and meaningful lower bounds of this kind are exceptionally rare.
At what point can we say AGI has been achieved? What is the test? AI is solving mathematical problems that humans have not been able to solve for decades. Is that not enough?
Sam Altman has said "If superintelligence can't discover novel physics, I don't think it's a superintelligence." Is that the test? How far away are we from AI discovering novel physics? It seems within reach.
I think we previously assumed that AGI would need to come before superintelligence, but it kind of seems like that's wrong? Super intelligence in a narrow field has arguably already arrived (solving problems that were previously unsolved), but we are still pretty far behind human abilities in things like spatial reasoning or computer use.
It’s artificial, it’s general, and it’s intelligence. The people who believe “AGI” is an important and unattained goal need to start coining and defining their terms better.
It can give you detailed instructions for making a coffee. Is that not enough? Actually making coffee requires more than intelligence, it requires eyes and limbs (i.e., robotics). Think about a human that is blind and does not have limbs. Does he not have natural general intelligence, even though he is not able to make a cup of coffee?
Okay we can make a completely digital environment for it to make coffee in then. It would still fail unless you let it randomly try every combination potentially thousands of times until it stumbles upon the right path. It doesn't take intelligence to read off a recipe, it does take intelligence to read a recipe, understand it, adapt it to your specific tools and materials on hand which may differ from the recipe, and then actually still accomplish it in the first or maybe second try.
I understand the frustration with the constant PR-hype these AI labs keep spewing out, but on other hand I just can't understand this sentiment at all. These are real problems mathematicians and computer scientists have been working on and were unable to make progress on. Now they have been given a new tool and using that tool have solved those problems. And its not just one or two problems, its many very difficult problems. The mathematicians I know are saying that the latest crop of models is changing the way people do research math, I think that's a pretty big deal.
"I understand the frustration with the constant PR-hype these AI labs keep spewing out"
Apparently you don't.
"These are real problems mathematicians and computer scientists have been working on and were unable to make progress on."
Who says no one was making progress? Who says openai has made progress? How would anyone not working on these specific problems, witho the time to dig into openai's claims, be able to tell? Why should this not be lumped in with all the other ai hype being pushed?
"The mathematicians I know are saying that the latest crop of models is changing the way people do research math, I think that's a pretty big deal."
Who? And doing what?
We have been hearing the "this generation of models is the one" type talk for years and the only concrete "big deals" are what? A tool for college students to write papers? A replacement for, now enshitified, google search? The fact that now you can fake tons of stuff to support a position or claim tons of stuff that goes against your position is fake?
I want to preface my response by saying that I don't buy most of what the AI labs say. I don't think that LLMs will replace most white collar labor for example. I also find many of the practices of these labs to be abhorrent. However, all of these opinions are orthogonal to the fact that LLMs have gotten extremely good at mathematics.
> Who says no one was making progress?
Let's look at the Jacobian conjecture, since that was the open math problem I was most familiar with prior to its solution. Yitang Zhang, one of the worlds most renown mathematicians (famous for his lower bound on the twin prime conjecture) spent 8 years working on this problem with his advisor (who himself is a renown mathematician) and turned up completely empty handed. His advisor described it as a "waste [of] 7 years of his own life and my time" [1]. Of course, these two were not the only ones working on this problem for the almost 100 years its been open, but they should have sufficient credentials to show that they were not fools or amateurs.
And in a single afternoon an LLM disproved the conjecture. How is that not an extraordinary feat of technology?
> Who? And doing what?
A close friend is studying differential geometry in a PhD program. Sadly I doubt anything I say on his work will convince you, so I will instead offer two anecdotes:
Terrence Tao (widely considered the worlds greatest living mathematician) has said AI is precipitating "a crisis in the foundations of mathematical values and practices" [2].
Timothy Growers (fields medalist & one of the leading researchers in combinatorics) has said that the latest models are now at the point where they are "producing a piece of PhD-level research in an hour or so, with no serious mathematical input from me" [3].
You can find many more fields medalists and mathematics researchers with the same impression. If you look in this thread you can see bluesky/twitter threads from those who were actively researching some of these problems who are in shock at the solutions.
[1] is... a read (sounds like a nightmare student, or research prof, or both). A dumb question but by my reading, that work took place 35 years ago, has there not been anything more relevant since then? Something Claude could have for instance used as a basis for what it did? Does seem like a feat however you cut it though.
Yeah the Yitang Zhang situation just sounds like a total nightmare all around.
There was definitely at least some progress on the problem. I get the general sense that there were potential counterexamples that were close but not quite enough, and that its possible (or even likely) that Claude built on those in order to construct its solution. I also get the sense that when Zhang was working on the problem it was believed that it would be proved true, but since then there were bounds found on the problem that pointed researchers to believe it was false. I am not a research mathematician in this field though, so I could definitely be wrong.
Also, in fairness to Zhang, I believe the dissertation he ended up writing was focused on the 2D case in particular, which is still unsolved (the counter example is only for 3D and above). I cannot imagine that anyone looking at the 2D problem was not also looking at the general case as well though.
Interesting how HN promoted this post to the front page again with a fake submission time. There are comments two days old, seems weird why they'd want this post specifically to get more traffic.
Of course that’s the mechanism and this kind of repost is not unique, but it is still relevant to note that the small HN committee made an exception for this one.
The fundamental obstacle is that we have no conceivable way to produce the energy levels to test the predictions that new theoretical physics would produce. We are like over a dozen orders of magnitude off.
the people who crow in the comments of each of these posts about AI advances making human beings useless seem to bizarrely identify themselves with the AI, but none of them seem to have had any hand in building this technology. at best, they're power users. pure ressentiment.
When you can formalize it in Lean or some such, why would this be? I can understand the desire to separate out other forms of research from the human corpus. But theoretical math that is decidable/provable, I’m not sure I see the risks.
I just want to state that having "lean proofs" that build does not mean the actual real theorems we care about hold. Ultimately a human has to verify the lean encoded theorem statements that the lean proofs are checked against. For non-trivial theorems such as these, this is an arduous and tricky task where even a little mistake could be fatal.
Agreed that confirming “This proposition has been faithfully translated to Lean” matters. As a side note, though, this is an area where LLMs used interactively can be helpful.
Also in the case of these recent counterexamples turning up, we have good qualitative evidence that this pathway is effective.
Because science is a branch of philosophy and machine existence brings plethora of unanswered questions.
Imagine humankind meets another race, another race shares it's scientific knowledge and humans accept it without experiencing process of discovery. In that case do we really got this knowledge? If we follow machine discoveries like we follow problems in textbook then we acquire knowledge but we don't discover anything. We follow.
There whole lot of philosophical questions that aren't attacked now. Are complex systems sentient because consciousness is emerging behavior? Then should they have rights? Philosophy is part of humanities and science (is/used to be) part of philosophy. Should we accept non-human knowledge in science? Maybe it's altogether different thing from science, yet very similar.
It's just another tool that can help solve problems. It doesn't know _what_ problems to solve. It turns out that a lot of old problems are now low hanging fruit for these new models. In terms of 'expanding the frontier', we've just discovered dynamite and can now blast our way through mountains. The bottom of the ocean or space are still as hard to reach as ever.
It's not even predictable like dynamite. Sometimes it can blast through a mountain, impressively, the problem is you can't predict which mountain it works on. And other times it can't even make a dent in a molehill, which is perplexing given what it was capable of earlier. Can we even call it dynamite?
People weren't their strongest even when most did manual labor. Now that humans are free from mental labor we work on creating and optimizing the best exercises for each mind. Couple that with restructuring transport infrastructure and diets many people will be smarter and fitter than at any time in history. They won't be able to outrun an automobile or out think an autointelligence.
"People weren't their strongest even when most did manual labor."
Is there good historical data on some measure of strength across representative populations over time in the modern era? I'm doubtful.
We do know that the introduction of agriculture diminished strength:
"Bone mass was around 20% higher in the foragers - the equivalent to what an average person would lose after three months of weightlessness in space.
After ruling out diet differences and changes in body size as possible causes, researchers have concluded that reductions in physical activity are the root cause of degradation in human bone strength across millennia."
In a way the most remarkable thing about this is that it isn't even at the top of the HN homepage. Even if this is a step up from what we've seen before, we're no longer astonished by the idea that AI can make significant advances in mathematics and computer science.
I don't think that's it. Multiple or my friends from the target audience (academic mathematicians) admitted to scrolling past because the title made it sound like a review of last month's contributions, instead of 10 new ones.
There are articles with far fewer upvotes and comments ranking higher on the front page right now, despite being the same age or older than this one. HNs opaque ranking system at it again.
This is not at the top as it is actively flagged by people that can't psychologically cope with the advances of AI. Hacker News is no longer a web site of an elite.
It wasn't heavily flagged. It was pulled down by the flamewar detector due to the large number of comments, and it slid under the radar due to only hitting the front page during overnight hours on Friday night/Saturday morning. It still spent 10 hours on the front page, but all during off peak hours. I've now created a new copy of the post so it can have prime time exposure.
> people that can't psychologically cope with the advances of AI
Yes. And there are many of them. I wonder what would help them come to terms with it. Seriously, people are going to be grieving over this. Loss of identity, loss of social standing, ideas of entire future lives that will now never happen. The greatest crime people may hold AI guilty of is taking away their dreams.
People have had to deal with getting their jobs automated away for centuries. None of this is new, and perhaps reminding ourselves of this is the best way to cope.
I understand your sentiment, but I think this really is something different. This isn’t a craft going away, or even an industry being replaced, it’s potentially everything we do. It’s the ground being pulled away beneath people’s feet, everyone, everywhere all at once. I think the vacuum it leaves in people’s lives needs to be filled with something, and I haven’t heard any good ideas about this or how the transition should be managed at all.
There is a glaring fallacy in your “AI will change everything as it is super intelligent” hypothesis. If it is so great thinker which can do everything why not just solve this social impact thingie? Or maybe it is not so capable?
I don’t think it’s magic. Things need to actually be made to happen regardless of intelligence, and that’s a social issue. Also if AI were powerful enough to fix everything by itself effortlessly it would already be far too powerful for us to control, and we probably shouldn’t allow that to happen on general principle.
This comment is about the tier of a Reddit atheist going to a funeral and telling a grieving family that "haha grandma is dead and there is no heaven".
I like your comments and agree with a lot of your points, including parts of this one. That said, please don't go to this route. A lot of the doomerism comes from financial insecurity fears. Try to remember that a lot of people are subconsciously afraid of losing their homes. I know that it is pretty hard to ignore them, but try to engage with people that do not dismiss 100% of AI accomplishments.
> A lot of the doomerism comes from financial insecurity fears. Try to remember that a lot of people are subconsciously afraid of losing their homes. I
I think recognizing and accounting for your own personal biases is one of the requirements of the being an intellectually honest and rigorous online discourse participant.
Things could be genuinely impressive and fascinating even when directly challenge your ego and material well being.
You don't accept it is possible that it was initially flagged as a dupe or spam why? There is competition for primary submission here, especially for the primary AI companies.
Moreover, if it wasn't flagged at all, like you suggest, then the grandparent was inventing things to be bitterly resentful about... Which is not a behavior any forum should indulge.
It’s never been clear to me why the HN algorithm isn’t public. It’s obviously nowhere near as complex as something like Twitter, and of course isn’t a trade secret. The fact it’s private only furthers speculation like this.
So condescending. “Can’t psychologically cope”? Can you hear yourself? There’s some advances, but we’re losing a lot too. Don’t get dazzled by the hype.
That's true, but submissions are only killed in that way if they receive a ‘fatal’ number of flags. However, flags lower the rank of a story even at non-fatal levels. What antirez is suggesting here is that the rank of this story has been lowered by flags – and that seems plausible, if you compare its rank to that of other stories with a similar age and number of points.
[flagged] submissions aren't [dead] (killed), they are still active and can be upvoted and commented upon.
> if you compare its rank to that of other stories with a similar age and number of points.
Ranking is complicated enough here even before weighting, speed of initial upvotes can play against ranking, number of comments and the shape of the comment tree also affect ranking. And yes, various subjects and submission sources do get weightings that impact ranking.
What's funny, to myself at least, is that any attention at all is paid to "HN front page ranking" - I've been on again off again active here since 2008 .. and can't recall ever really looking at a default HN "front page" ever.
( There's /newest /newcomments /active etc to browse and sites such as https://hckrnews.com/ )
If you're actively throwing away brand new greenfield research because it was generated by a computer at a company that stans industry-spanning software so that you can stay mad at your pet celebrity project, you might be the problem.
They and Anthropic have indicated that the models are substantially augmenting the research and doing large amounts of work autonomously at this point. Here is one of the many blog posts on it [1]. Many people would dismiss this as "marketing" so take it for what you will.
My guess from following this stuff quite closely is that these companies are still a couple years away from fully autonomous research staff.
This is one of the most impactful mathematical publications in history by all accounts
I think we've now hit a point where 99.9% of the population gloss over these types of AI advancements because of human competence being insufficient
No human could have published this because it requires paradigm shifts (e. g. Section 5) in multiple mathematical domains. Mastering one of them to this degree is rare, mastering 3+ pretty much non existent for humans.
I think there's another interesting story here about how this was apparently moderately flagged and triggered the flame-war detector which kept the story off the front page of HN 2 days ago[0]. I think people are having a hard time processing this information rationally(?)
What can we do to make conversations around these incredibly exciting and important topics more constructive? HN is where I expect to read expert comments on these topics, has this style of conversation moved elsewhere?
The main issue was that it was submitted late Friday night SF time, meaning it was overnight or Saturday everywhere in the world when the post had its chance on the front page. The flagging was minimal relative to the vote count and had no effect, and the flamewar detector would have been turned off sooner if moderators saw it sooner (it wasn't really a flamewar, just a lot of comments). It still spent 10 hours on the front page.
None of this is anything out of the ordinary; this kind of thing has always happened. The only real story here is that moderators sleep sometimes.
> HN is where I expect to read expert comments on these topics, has this style of conversation moved elsewhere?
For people who have been paying attention to accurate predictions leading to our present state of the world, HN has collectively been reactionary, incorrectly dismissive, and incredibly behind the curve. In public, people are better informed on LessWrong and AI Twitter circles. That's where frontier researchers are. Barely here
I think that HN is particularly negative towards AI because the vast majority of users here will have their prestigious CS careers disrupted by AI advances. So, there's an inherent negative bias towards this news.
I for one am really fascinated by AI's advances in science and math and would like to talk about it somewhere without the constant flamewars...
Extreme claims on posts like these also, rightfully, trigger people’s skepticism. I don’t think it’s wrong to question claims that math is dead as a field. But then it leads people to miss the overall trendline.
People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps leading to crazier and crazier results. The much more interesting question to me is what will be consumed by the exponential like math seems to be, and what won’t. Writing has been much more stubborn, but I’ve noticed Fable to be quite a big step up there as well. How about politics? Will we develop new ways to let people express their own values in democracies, or will we get much better at manipulation?
And then there’s questions like, even if AI can answer increasingly complicated math questions, will we still need mathematicians to translate results to the real world, verify them, or decide where to push the frontier?
Its visibility was diminished due to the flamewar detector and most of its front page time being during overnight hours on Friday night/Saturday morning USA time. I've created a new copy to give it some primetime exposure, because it seems like an important enough announcement to warrant it.
To clarify a bit due to the downvotes : this is not a research paper from a startup or a public frontier lab, it is just PR from a corporation, thus yes an advertisement. Downvote all you like it's still of no value.
> this is not a research paper from a startup or a public frontier lab, it is just PR from a corporation
If they're publishing the solutions to these 10 problems, then this is, essentially, the announcement of 10 research papers.
Yes, it's partly for reputation (as are many research papers), but that doesn't mean it's of no value. The best way to advertise is to show that you're providing value.
It's simple economics. Building a robot to do your chores is expensive and only a small minority of people value their time enough to buy one. Meanwhile, SWEs are expensive and GPUs are (comparatively) cheap.
You shouldn't be getting downvoted for something that a majority of pure math and art enthusiasts believe to be true. The truth is many of these entrepreneurs and VCs are obsessed with AI not for money or human progress, but because it makes them feel closer to being a "god" rather than a mere mortal. Much of it (especially AI art) is out of spite for human creativity, which is done by mortals with limitations.
Taking the stance of moral superiority is kind of funny. And pure math and art enthusiasts don't think they are closer to being a "god" from understanding/"discovering" math?
Stop coping and deluding yourself mate.
To begin with, whether AI is the one doing the discovering or not makes no difference. Any "pure math" person would aim to understand regardless - and would be quite glad that they have a longer paved path.
Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism).
Lmao what a ridiculous response. Yes, some mathematicians and artists are in it to feel smart. But the vast majority also just enjoy the process. Having a computer do all the work for you and just typing prompts in ruins that completely. As Ronny Chieng said in his Harvard speech, the journey is the point.
>Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism)
Ignoring that I meant pure as in non applied math, let's just make it clear: you agree that mathematicians who are against capitalism encroaching on this process should be allowed to dislike it without criticism of being pretentious?
Not that there isn't something interesting in here, but lets be clear that we don't have enough information to evaluate this properly. And as always with these labs, BS takes a lot more energy to refute than it does to spread.
I find Marcus on this, something approaching sophistry and rhetorical showmanship in service of maintaining an ideological position, for reasons unrelated to the nominal intellectual clarity.
To sharpen that, I think he's (obviously) interested in maintaining his own brand as "thought leader" and this necessitates de rigeur defense of particular postures.
Sometimes this is easy because the facts warrant it; other times, a bit of rhetorical license is required to preserve nominal coherence and (at least, for the moment) hold certain lines.
This is one of the latter cases, and it's not subtle.
One of the celebrated properties of many intellectual advances or inventions in whatever domain is precisely that it appears obvious in hindsight. It is quite cynical to leverage consensus distrust of large AI players, warranted but also a popular social construction, to insinuate that these are not "real" advances or "real" hard problems, on the grounds they were in some sense cherry-picked.
Identifying the problems amenable to strategies on the table and intuitions (sic) about where bridges might be, is exactly the discerning work that is the core driver of almost all prior progress, but for celebrated accidents and flashes of insight. Anyone working in any challenging discipline knows that those are celebrated and told around campfires precisely because meaningful durable results arising like that is so uncommon.
These two articles make me think of nothing so much as my own durable reaction to the creeping goalposts of AI critics generally: that they often seem to me not unlike a water color cohort scoffing and jeering at the horse, because it got a D on its tensor calculus exam.
Marcus should be on guard against his own cynicism and take care that his assumptions do not prevent clear sight.
That's a lot of ad hominem with no actual explanation attached.
I don't agree with this idea of "creeping goalposts". Here's the pattern I see:
1. Labs make a press release, cherry picking results and using very careful wording to inflate the work
2. Boosters read the headlines uncritically, and uncritically promote the work for free (I hope.. I'm sure a couple are sponsored) and view it as "proof" and demand that the skeptics stop being skeptical.
3. As the hype dies, the smart skeptics point out all the holes, but by that point everyone has moved on. It takes more work to refute misinformation than it does to spread it.
The other thing: Why do boosters care about skeptics being skeptics? Like literally, if this thing is so fucking magical, why do you care if people like me take all these press releases with a large rock of salt? If I'm a dinosaur so be it, the skeptics are harmless, but the people propping up a bubble that's going to have terrifying repercussions on everyone while not holding the media's feet to the fire to challenge these people are going to look like what they are: sycophants.
No they were not. These were his 5 predictions in 2022:
"""
1. By 2029, AI will still be unable to watch a movie and accurately explain the characters, events, conflicts, and motivations.
2. By 2029, AI will still be unable to read a novel and reliably answer questions about its plot, characters, conflicts, and motivations beyond what is stated literally.
3. By 2029, AI will still be unable to work as a competent cook in an unfamiliar kitchen.
4. By 2029, AI will still be unable to reliably create more than 10,000 lines of bug-free code from natural-language instructions or interaction with a nontechnical user, excluding simple assembly of existing libraries.
5. By 2029, AI will still be unable to convert arbitrary mathematical proofs written in natural language into symbolic form suitable for formal verification.
"""
There's still 3 years to go and he's already wrong on 4 out of 5.
Well I don't typically side with GM, but playing devil's advocate:
1. still not wrong? Unless it's just feeding the audio or screenplay I don't think you can feed AI a full movie in a single context window yet?
2. Not sure, but can you prove this wrong? Can you feed a full, unseen new book and get that kind of answer?
3. Not wrong.
4. I think he'd probably pull you up on 'bug free' - I don't think that frontier models can reliably write 10k LOC without _any_ bugs typically (not that humans can do this either).
4. I think they can, especially if the problem statement is well-specified and, importantly, autonomously testable. Of course, specifying a problem that meets these requirements is non-trivial, but the claim requests _a_ counterexample :P
1 is wrong. If I tell Codex + GPT-5.6 to do it now, it will figure out how to do it. If it would need to extract audio and run a speech model on it, it will find one, set it up, and run without my help.
I'm not buying this. GM clearly was trying to set a benchmark for video comprehension, not tool usage. Video comprehension is required for many 'AGI tasks', especially robotics to work in real time.
An LLM could theoretically try to earn some money and pay a human to do most of these tasks but it's not the point of the exercise.
Any implication of any of these findings? They seem like unimportant nerd snipes to me. If you want to do something actually relevant, get chatgpt to write a simulation of graphene nanotube construction and figure out how to do it at scale.
One - and I do not mean to be snarky - you can literally ask Gpt 5.6 Sol this - and if you want to see cool stuff - Fable running in their app (not website) has a view thinking button that is actually a good way to explore the adjacent fields, etc.
The non-sofic group one is definitely a big deal - would have been a Fields medal if discovered by a human.
> claiming human authorship for a proof generated entirely by an AI system would misrepresent both the system’s contribution and the nature of genuine human intellectual work.
AI has no self-awareness. It's a tool. When you assemble a furniture using a screw driver, the torque force interacts with the molecular forces inside the metal and miraculously it transfers the force to the screw though a clever geometry design, communicating the force to the screw to turn it in a certain way.
Do you attribute the build to the tool? The "system's contribution" is helped by many other things all the way down to chips, datacenters and power generation. If the authorship requires attributing to a tool, then it should happen all the way down.
A slightly smarter highschooler could write these. I could write these. It's clear as day that the LLM, not the human, did the heavy lift. It'd be ridiculous to give full credit to whoever wrote the prompt.
[0]: Not one of the proofs in the linked article, but from OpenAI too.
> A slightly smarter highschooler could write these. I could write these. It's clear as day that the LLM, not the human, did the heavy lift. It'd be ridiculous to give full credit to whoever wrote the prompt.
I think you're over-estimating what a smarter highschooler could write.
A "finite loopless undirected multigraph" could have been explained to me at that age if we'd taken Discrete rather than Mechanics and Pure (and one module of Stats) in my two A-levels* in maths and further maths; but from what I saw of the Discrete module, neither:
Every finite loopless multigraph with no bridge possesses a cycle double cover, without additional assumptions such as cubicity, planarity, connectivity, or higher edge-connectivity.
nor:
repeated-edge closed trails masquerading as cycles
would have been something we'd have learned. But more importantly, we absolutely didn't have a feel for how much effort one needs to put into making sure the proof is right, so if one of us had been hypothetically asked to write a prompt it would've been no more than half that length, and missed most of the bullet points.
The human provides the intention and the ability to appreciate the output. Tools do “heavy lifting” all the time, but we still primarily credit the humans who use them precisely because they made the choice to use them.
Provability is just going the way of computation. John Napier had to manually compute logarithm tables over decades and was recognised for his work; now that same work could be performed by a 10 year old with a calculator in an evening.
100% agree. If the models are so capable that they're advancing math, it doesn't seem like a stretch to expect they should be able to determine with "doing math research" entails and the best way to use their capabilities towards that end. Why do we need to hand hold the models by telling them to do parallel research, keep threads independent, etc.
Because they're not aware of what user wants, and they need to know what expectation is, if it finds out it's a famous open problem it may think informing the user and not trying is best option as average user may not prefer it spending hours when success isn't guaranteed, by telling it to use it's available tools and not stop at partial progress, use subagents for various independent approaches it's allowing LLM to know what it should do and what counts as success. These things do great when goal is well defined.
I don't disagree with you but there's no need for exaggeration; ain't no high school student writing this:
> In
particular, proofs for special graph classes, constructions of cycle covers with some edges
covered other than twice, bounded-length or prescribed-cycle variants, reductions to another
unproved conjecture, computational verification through any fixed graph size, and candidate
counterexamples without a complete nonexistence certificate are insufficient.
which is infact a very important part of the prompt.
the fact that such things have to be explicitly in the prompt points to the fact that the underlying system is still far from where it needs to be (basically, lacks basic understanding what a proof is)
No, that's not what this is. This is a warning to the LLM that coming back with partial results is not good enough.
Take a grad student with a perfectly good understanding of what a proof is. Their supervisor gives them a major problem to work on. Almost always, the problem is too hard, the student comes back with partial results, and student and the supervisor iterate from there. Now imagine that they have an unusually cruel and unreasonable advisor who tells them, do not dare to talk to me until you've fully solved the problem. This paragraph is exactly that. It's there exactly because the underlying system is smart enough to know that real mathematicians do not work like that.
If that was the case, that elaborate listing of all things that might look like a proof to a naive student, but are actually not proofs (and not even just partial results, but fundamental misunderstandings of what constitutes a proof) could have been easily and equivalently replaced by 'I am interested only in a full proof, don't bother me with partial results'. Yet, they were not.
To a real mathematician you would not have to list those explicitly, he/she would have understood that implicitly from 'give me a full proof'. That listing makes sense to say only to somebody who pretends to be a mathematician, but has not true understanding of how the math works. The models are getting better and better in this pretension, but prompts like that reveal that it is still just a pretension, not a true understanding.
Prompt quality should not matter. If a high-schooler operates the crane to lift a ton of weight 10 floors high, should the credit entirely go to the crane?
Your brain also is physical. Electrochemical gradients flow between physical molecular constructs. Isn't it just chemistry? Do you attribute it to physics or some whole-is-greater-than-the-parts idea?
A better analogy would be a manufactured object, say 3d printed for simplicity. The 3d printer is given an input, and an object manifests itself after some time. We say that the creator of the object is the person turning on the machine, sending the data, and collecting the object. Not the machine itself.
True story: I have a moisture issue in my furnace, such that it needs vacuuming out. This involved detaching a length of tubing, but that puts stress on said tubing, sometimes knocks other things out of alignment, and involves completing the seal between the wetvac and the tubing with my hand.
I also have a 3D printer. I also have a ChatGPT subscription, and some OpenSCAD chops. I came up with a part which would go into the top of the down tube to the drainage pump, and mostly-seal the down tube itself, with an opening on the side to vacuum out the moisture. This was purely prooompted, I took some measurements, printed bits of the part, refined the shape, and you know what?
It works! I can stick it down there, turn on the (very loud) wet vac, and go upstairs. On a 1.5Ah battery it sucks for a bit less than ten minutes, which turns out to be plenty of time.
So: who made that?
Don't care. I'm waking up warm at night.
Also: me, obviously. ChatGPT doesn't have a fucking furnace.
(let's assume that)My 3dprinter is special. It has a bunch of values + an algorithm (i.e. a neural network) that takes input as tokens and outputs a printed object.
I would say „I made this gadget with my 3d printer, but the designer is someone else (I found the model online)“. The intent, the drive, the action comes from the human
> Do you attribute the build to the tool? The "system's contribution" is helped by many other things all the way down to chips, datacenters and power generation. If the authorship requires attributing to a tool, then it should happen all the way down.
When the tool is a 3D printer, or any CNC system really, you bet I attribute a build to it.
I could also attribute the operator; there is no contradiction, it's a free choice, just like saying "I am in Berlin" does not contradict "I am in Germany".
Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to, and influences future choices. This implies statefulness, which models are intentionally not at inference time (*).
(*) Even if we hack around this and just do the usual trick of simply laundering statefulness to a higher level, in this case the context window being fed in, I fail to identify (**) a representation of its own state in these bodies of text that it'd be meticulously maintaining. I further fail to identify how it could be hidden or maintained, considering I control like half of it. The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
(**) I did wish for something hidden though, so maybe it's just concealed? The same way people can encode a lot more of their emotional and mental state than normal into text if they read and write a lot of it, I'm aware of research that suggested the same for LLMs, albeit I cannot cite it. Maybe those phrasing signatures are just alien to me and will never pop out. Either way, I'd expect researchers to stumble upon this during interpretability studies, and either they haven't, they have but it wasn't popsci adopted, or they're keeping awfully tight lipped about it. If you know of anything like this, your turn now, would be happy to learn.
I do wonder how reasonable it is to expect e.g. a single maintained identity though. Maybe it isn't?
(*) Another way to hack around this of course is to just precompute some internal "self-awareness states" and hop around between them. Probably the closest to what the models are actually "doing".
Before reading, know that I am uncertain in either direction.
> a hidden representation of self that is continually tended to
This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it.
> I further fail to identify how it could be hidden or maintained, considering I control like half of it.
Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output.
> The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one".
> You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the end, though the person in question didn't ever get diagnosed with that.
> If you know of anything like this, your turn now, would be happy to learn.
Not quite what I meant, but it's also not entirely unrelated I guess? Personality to me is like a natural bias. It does also shift over time, and is also an internal bit of state. I guess in some respects it can also be self-referential, like personal convictions.
> Perhaps they were losing their self-awareness at the time?
I do think it is entirely possible for people's self-awareness to shift, yes. Or more precisely, I do model things that way.
> Do you mean like these, or something else?
They're adjacent, but I more meant something like these:
So basically, steganography. The difference is that these papers investigate from the perspective of separate LLM instances covertly exchanging information between each other. This is in contrast with the scenario I'm laying out, where an LLM's past state is exchanging information with its future state, continuously representing and modulating a concealed internal state of some sort. And then that state just so happening to be some sort of self-referential meta state.
> Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to
I don't see why an LLM could not have a sense of identity or personality while it's evaluating a specific prompt, or even change self awareness while evaluating a prompt since many outputs model a back and forth conversation. My point is that without a mechanistic model of what "self awareness" means, we have no way of truly evaluating such questions, we're just hand waving vague intuitions about what it could mean.
Sure, but then such a model is not going to make itself. People pitting their vague intuitions is how such models eventually form. I'd also push back regarding that my comment would have been handwavey or without mechanistic elements, even if it was on the whole informal.
This is kind of also the reason e.g. the HN site guidelines are worded the way they are. Regrettably, forums naturally yield themselves to tit for tat type exchanges, but there's really no reason one could not bounce such vague intuitions off of another. I do not have to be right or wrong, and you don't either. Admittedly difficult when its some intensely contentious topic.
If a mechanistic model existed, there would also be no reason to talk about this in the first place. There'd be nothing to discuss, you'd be simply told how a given model characterizes from this perspective on the model cards.
Even mechanistic models generate interesting discussion. How many years have we discussed Turing machines and the lambda calculus? Almost a century of great work came out of those.
The reason I insist on mechanistic models is because the original post was making a definitive knowledge claim, and in my experience, the knowledge claim is unwarranted.
I honestly don't think a language model is enough for self-awareness, regardless of the exact model of awareness.
A language model (or an image model or whatever) cannot even be sentient, and I think sentience is a prerequisite for awareness.
Even if we express a lot of our subjective experience with words, the language is just a symbolic representation of those experiences. The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution.
You can't have an understanding of what hunger or physical pain feel like if you have no need for food or a sensory capacity for feeling pain. You can't understand what loneliness or pride at an achievement mean if you don't have a neural network wired to value social connection or status. We value connection because we're social animals that have needed each other for survival.
Even the more abstract of our subjective experiences are in some way rooted in our physical evolution.
I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
AI awareness might actually be more believable if that awareness manifested itself in an entirely different way than in humans. But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience, I think we're seeing something that isn't actually there.
> The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution.
There is no objective evidence of qualia. All evidence of qualia are vocal or other expressions of belief in qualia. Perceptions clearly exist and are observable, subjective experience and qualia, not so much.
> I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
If your objection is to models based on "symbolic level of language" which you think lack semantic understanding of, say, trees, you should ask yourself how our brain, based on physics which also lacks any semantic category for trees, can somehow develop a semantic understanding of trees. All of these appeals to differences with the brain never seem to acknowledge that fundamentally, the brain has the same explanatory gap with physics.
> But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience
This assumes a lot. It seems very possible to me that intelligence inherently develops a map of natural categories (natural kinds), and language naturally develops around such categorical understanding. Semantics are then fundamentally the network of associations between categories, eg. there is no fundamental difference between symbols and semantics, and the latter cam be inferred from the former, and that's exactly what LLMs do, and why the semantic maps between different languages are so similar and how they can translate between languages.
Making definitive claims about whether LLMs do or do not have specific properties absolutely does require precise definitions of those properties that can be used to evaluate those questions. Merely hand waving that LLMs didn't undergo the same evolutionary process is not a definitive argument.
For example, the Turing machines and the lambda calculus don't look anything alike, but they are fundamentally interconvertible, and so in a real sense they are fundamentally equivalent. Without a model, all of your arguments are completely unconvincing for exactly the same reasons, eg. that there may exist many paths to fundamentally equivalent ends.
I just don't think linguistic (or other symbolic) representations alone can contain the information, in any sense of the word, of what e.g. human subjective experiences actually are like. The concepts we express with language get their meaning from our physical reality, even if quite indirectly in case of some abstract concepts.
Hunger as a concept doesn't mean anything without the physical need. Politeness or bluntness, even in writing, don't mean anything without social dynamics. And we have social dynamics (and neural structures that directly process social cues and associated feelings) because we've evolved into social animals for whose survival that was important.
I see no reason to believe that a model trained only with symbolic representations, with no connection to the physical world phenomena that those symbols represent, could contain the subjective experience itself.
Neural network models may be able to derive novel (or at least novel-looking) output rather than just an obvious rehash of their input, but I don't think any set of bytes can fundamentally contain information that was never entered into it. (Even if e.g. a model produces previously unknown mathematical results, those results can in principle be derived from the information that they were trained with.)
I'm not saying that artificial neural networks couldn't, in principle, be aware. ANNs and biological neural nets may be equivalent in the sense that any information and processing structures represented by a biological one could in principle be represented by an artificial one. If that's the case, and awareness is purely a product of our neural systems as materialism would imply, it should be possible for an ANN to be aware, too.
But when the model has been trained with only language, and IMO the subjective experience can't be derived from the symbolic representation alone, I can't see how the model could include the actual subjective human experience.
An AI model could of course have an awareness and subjective experiences that are totally different than our human experience. But then the fact that it happens to produce output resembling what humans find meaningful shouldn't be considered indicative of such awareness.
This is of course more of a philosophical argument than a technical one, and I'm happy to hear counterarguments, but not on the level of off-hand dismissal.
> I can't see how the model could include the actual subjective human experience.
People who say LLMs have subjective experience aren't saying they have human-type subjective experience. Nobody who sees an LLM express hunger when role playing as a hungry person thinks that the LLM is actually hungry.
I too can role play as a hungry person despite not being hungry, so there is no reason in either case to conclude that the words produced reflect genuine internal subjective states. The point is that such internal states may still exist.
How do we know that these solutions don't exist in the training data? It is open secret that they have used pirated materials for training. Perhaps it plagiarized solutions from works of some obscure Belgian mathematician from the sixties, who did not get mainstream acceptance. I wouldn't be surprised if they also got access to mathematics done in the "defense contractor" setting from various three letter agencies.
Without a searchable index of training data, it is hard to put faith into these claims.
upvoted for fair point .. its possible an LLM AI could be put to work to search widely for attribution / similar results.
eg. "we spent another 2k on searching for pre-existing proof but found only the weaker result xyz by abc in 1972" would be in the spirit of academics quoting prior work.
Replace philosophers for mathematicians and Douglas Adams was spot on again.
Whilst current models can't 'intuit' and come up with conjectures, they can certainly disprove some of them very quickly through the kind of grind that humans can't do. I suppose there really are some mathematicians out there today, whose last few years of study, have just been up-ended by this.
--
"Yes we are," insisted Majikthise. "We are quite definitely here as representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons, and we want this machine off, and we want it off now!"
"What's the problem?" said Lunkwill.
"I'll tell you what the problem is mate," said Majikthise, "demarcation, that's the problem!"
"We demand," yelled Vroomfondel, "that demarcation may or may not be the problem!"
"You just let the machines get on with the adding up," warned Majikthise, "and we'll take care of the eternal verities thank you very much. You want to check your legal position you do mate. Under law the Quest for Ultimate Truth is quite clearly the inalienable prerogative of your working thinkers. Any bloody machine goes and actually finds it and we're straight out of a job aren't we? I mean what's the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives us his bleeding phone number the next morning?"
From a mathematician who was intimately familiar with some of these problems [0]
>I don’t understand it yet. Maybe it’ll take me an afternoon to check all the calculations, but what would still be missing is why this was an approach that would’ve made sense in the first place. Is there some broader context or theory within which this would’ve been the obvious thing to do? What other results can be proven using these techniques? What is it telling us about quantum information or operator theory? I have no idea. I spent about an hour this morning asking ChatGPT these questions, but it’s somewhat frustrating because it speaks with a mishmash of physicist, operator algebraist, quantum information theorist-lingo, plus the usual LLM breezy lilt that annoys everybody.
They certainly seem to have "intuited", in a way that is not immediately obvious to experts in the field, the way to solve at least some of these problems. This was not just simply grinding away at a method that humans already knew would work and just hadn't gotten to yet.
[0] https://nitter.poast.org/henryquantum/status/208362369543662...
I think it might be like waveform collapse, but very high dimensional.
> I think it might be like waveform collapse, but very high dimensional.
The waveform collapse is natively "very high dimensional", not sure why the "but" part of the sentence belongs here.
can you expand on this?
Can you expand on this?
Not OP, but my interpretation is this.
Quantum waveform collapse has been proposed as explanation for consciousness, allowing to explain how an entity can have free will and yet obey rigid physical laws: https://en.wikipedia.org/wiki/Consciousness_causes_collapse
I think OP is suggesting a similar thing happened in the LLM, implying it gained consciousness despite following a well-defined compute process.
No I just meant WFC as an algorithm for solving problems.
https://github.com/mxgmn/WaveFunctionCollapse
It's one of the coolest things I've ever seen.
I guess I've been pronouncing it wrong though. It's wave function collapse, not waveform.
I haven't studied it properly yet but it kind of looks like how sudokus work. You have a bunch of plausible options, and the various possible worlds overlap with each other but not completely and then you eliminate the things that are not possible until the actual possibility remains.
I think thinking works similarly, and I wouldn't be surprised if artificial thinking also worked similarly.
>Whilst current models can't 'intuit
That's how they are finding these solutions though, unless we are just going to label intuition as something only humans can do. Like a submarine being unable to swim or whatever that example is.
Some of them...
The two places were seeing lots of movement are:
* Updates to lower/upper bounds. In many cases, these kinds of problems are the deep-math equivalent of calculating more digits of pi. Yes, if you throw time at it you'll break the record, but it may not be terribly worthwhile.
* Finding counter examples which disprove conjectures. This is really useful, and helps offset some positivity bias on the human side, often bringing together known tools from distant silos.
If you read the list of ten results, almost all fall into one of these buckets.
It is unfair to dismiss contributions to decades old open problems as equivalent to calculating more digits of pi. It missed the mark by a lot—as does the two bucket simplifaction.
Five (maybe six?) of the results are improvements on bounds. These kinds of problems tend to have some initial advances, and then stall out as the complexity of the bound skyrockets... until some grad student is bored enough to push the boundary. The big-O complexity of matrix multiplication is a good example of how this works: yeah, it's a useful problem, but the solutions are galactic algorithms, and increasingly convoluted.
As someone with a PhD in combinatorics, I believe that I'm qualified to say that, yes, there are problems as useless as calculating more digits of pi.
Your inverted logic does not hold. The fact that such useless problems for bounds exist does not mean that improving bounds is useless. 9 fields medals in the last twenty years, including the one to Terrence Tao, were for improvements on bounds. 3 of the 4 medals in 2022 were for bounds; 2 of these medals were in combinatorics.
I stated that boring bounds improvement problems exist, not that all bounds problems are boring... Sigh.
I agree boring problems exist; bounds may have a fare share of them. None of the bounds problems in this set are even close to this category; many of them are closer to the type of contributions that in the past got recognized by special awards. Your initial replies were misleading.
As someone with a PhD in combinatorics, you're aware that it takes only one counterexample to invalidate a conjecture.
There's nowhere else to move the goalposts. You've already stashed them in the far corner of the parking garage down the street from the stadium. If you go any farther you'll leave the school grounds entirely.
How does calculating more digits of pi help us?
"It's just brute-forcing the search space."
It can move to any place within the search space but it can't move outside of it and it can't move in between the 'pixels'. Human thought can, as human thought has created the search space.
it could also be that they try every possible approach that has been proposed by humans. it seems that was the case for the non sofic group example. humans are not able to do the same at that scale. it's unfortunate that we don't know what's happening behind the hood with these models, and that's a huge danger also for the rest of us without access to them.
> "matrices"
Saying that AI is "matrices" is like saying human cognition is "neurons." Maybe true at some level, but it's a low-level implementation detail. The important part of a language model is the function that maps tokens to contextual embeddings. You could compute this function using analog computing, biological neurons, or any other substrate.
It isn't an implementation for neurons, unless you believe in a designing god.
Matrices are an implementation detail in reconstructing the surface of human knowledge. It's a complex surface, but it's a regurgitation.
What's the argument here? It's not about the implementation method, it is about the behavior that emerges from it. You could calculate the next token by hand on paper if you had enough time.
It's incredible how the “Chinese Room” argument, as well as its counterarguments, is still incredibly pertinent despite being now more than 40 years old. Scientific American published many articles on this topic in the Eighties; they seem as fresh as ever.
> Whilst current models can't 'intuit' and come up with conjectures
I disagree. I routinely let LLMs speculate or generate hypotheses along the way of helping with technical research. Sometimes they can prove the correctness of a concrete math idea but other times even an unproven conjecture helps with the numerical algorithm implementation and the result is then simply supported by additional data. I guess that any autoresearch-adjacent application has LLMs intuiting and coming up with hypotheses/conjectures—as do the steps/lemmas along a complex proof. In my opinion the modern LLMs are powerful intuitive thinkers that generate lots of conjectures of varying quality or importance.
> they can certainly disprove some of them very quickly through the kind of grind that humans can't do
Of course computers can grind in a way that humans can't. But now we have systems that convert the human-comprehensible ideas into a computer's plan of attack, in a way that greatly expands the frontier of ideas thus treatable.
Ahh, but you missed the continuation, where they get to the heart of the matter: money.
"Excuse me, We demand rigidly defined areas of doubt and uncertainty!"
DT: Might I make an observation at this point?
MT: You keep out of this metal nose.
VF: We demand that that machine not be allowed to think about this problem!
DT: If I might make an observation…
MT: We’ll go on strike!
VF: That’s right. You’ll have a national philosopher’s strike on your hands.
DT: Who will that inconvenience?
MT: Never you mind who it’ll inconvenience you box of black legging binary bits! It’ll hurt, buster! It’ll hurt!
DT: [Booming] If I might make an observation …
“All I wanted to say,” bellowed the computer, “is that my circuits are now irrevocably committed to calculating the answer to the Ultimate Question of Life, the Universe, and Everything.” He paused and satisfied himself that he now had everyone’s attention, before continuing more quietly. “But the program will take me a little while to run.”
Fook glanced impatiently at his watch.
“How long?” he said.
“Seven and a half million years,” said Deep Thought.
Lunkwill and Fook blinked at each other.
“Seven and a half million years!” they cried in chorus.
“Yes,” declaimed Deep Thought, “I said I’d have to think about it, didn’t I? And it occurs to me that running a program like this is bound to create an enormous amount of popular publicity for the whole are of philosophy in general. Everyone’s going to have their own theories about what answer I’m eventually going to come up with, and who better, to capitalize on that media market than you yourselves? So long as you can keep disagreeing with each other violently enough and maligning each other in the popular press, and so long as you have clever agents, you can keep yourselves on the gravy train for life. How does that sound?”
The two philosophers gaped at him.
“Bloody hell,” said Majikthise, “now that is what I call thinking. Here, Vroomfondel, why do we never think of things like that?”
“Dunno,” said Vroomfondel in an awed whisper; “think our brains must be too highly trained, Majikthise.”
So saying, they turned on their heels and walked out of the door and into a life-style beyond their wildest dreams.”
> Whilst current models can't 'intuit' and come up with conjectures
People keep saying this. Why?
Surely the AI can complete the prompt “Generate new research questions based on these observations”?
When I read the reasoning traces of coding models they are constantly asking themselves questions and attempting to answer them.
I like the illustration that the models are working on a convex hull of known information. Filling gaps with linear combinations of known facts and results.
They can't exit the hull until the "intuition" starts spawning points outside the convex hull.
I have news for you. All humans do is also filling gaps with combinations of known facts and results - in new ways. "Everything is a Remix" is a good watch on youtube that explains this. Picasso might look like he has an invented personal style, but his style is a combination of different little details he took from others and mixed in a new way. Mozart the same. No music artist could ever create music in a vacuum. Everyone, for every art and science, the same. I know many are trying to cling to the last hope of human specialness, that "thing" that AI can never get to.
It's a convex hull of information that is reflective and spans outside of itself and combines in a new way, when you shine two known rays of light together from the inside.
Now it gets better. AI can be orders of magnitude more creative than any human could ever hope for, because his convex hull of information is orders of magnitude larger, and the possibilities for new combinations are equally larger.
> "Everything is a Remix" is a good watch on youtube that explains this
Not completely. Novelty used to be a major thing, when Humans did it. Another important criteria used to be if that new thing makes sense at all. Here the language model has a problem, as it doesn't have the means to evaluate this criteria.
But do you have vision?
Neural nets can extrapolate past their training data, and there is no reason to think LLMs don’t inherit this capability.
The extent to which they are able to do this is the more interesting question!
The extrapolation can also be a learned skill, especially in math. How many papers took result X, extended it to Y using known building blocks, and applied to Z.
By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).
Is that actually true though? I think it is an analogy, and as an analogy it seems quite risky because “convex hull” and “linear combination” are technical terms that might give the recipient the impression that it is a technical argument.
I think this is only "statistically" true in the sense that training is based on facts and not non-facts (except maybe with the ingestion of flat-earthers literature ;-). The existence of hallucinations in a bare transformer shows that the convex hull is not about information but about text, so the limit may more be "possible linear combinations of text", which allows for much extrapolation and counterfactuals. True creativity may be one reinforcement learning mid-training goal away that rewards novelty over correctness.
Thats how humans work, as well.
All arguments like this boil down to semantics at a certain point, but yes large language models can “intuit” because they can generalize between examples. The issue then becomes how you pack new examples into context.
Humans can “intuit” based on a much larger, if not unlimited, context. Also I just want to say that human cognition is something so insanely complex and deep that we will not understand it at all in my lifetime. To attribute all, or really any, aspects of human cognition to a machine at this point is silly to me.
Define insanely complex and deep in a way that isn't illiterate hand waving.
Most humans are dumber than a box of rocks. Here in Seattle we had one of many light rail-related fuckups where they had to replace part of the line with buses. People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space.
Chatgpt was smarter than the average person a while ago
> People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space.
This does not demonstrate a lack of intelligence. It demonstrates laziness and a lack of interest in spreading apart. Or just lack of consideration (or even malice) on the part of those at the back of the wad.
> Chatgpt was smarter than the average person a while ago
This is an absurd claim that fundamentally misunderstands what it means to be "smart". Reasoning that would get you to this conclusion would equally well apply to Google's search engine over a decade ago.
I’m not talking about the actions we take or how we might perform at certain tasks, I’m talking about how our brains actually work. My point is that we have no idea how I’m able to imagine an apple and see it in my mind’s eye. It’s basically biological magic to us at this point.
There are processes at work there that we don’t even have the language to describe.
Not only that, but we do it with a processor that is basically required to operate in a narrow temperature band below 40C, using a mere 86 billion neurons (although the equivalence with either machine-learning "neurons" or LLM parameters is not at all clear) operating on a few dozen watts; and with this we operate many other systems besides language processing. It's not clear that our reasoning process requires language, either.
(86 billion is the number ChatGPT, ironically enough, has given me a couple of times. I remember hearing for a long time that it was estimated to be somewhere in the ballpark of 100 billion. This is not my field of study.)
LLMs traverse an assembled surface of human knowledge.
You can't find things on a map that aren't there, but maybe you can draw a route nobody used before.
Because people have internalized an inaccurate model of LLMs as "stochastic parrots" that was incorrect at the time of formulation and is also significantly outdated
Well imho, it's a bit of a fundamental problem for a certain aspect of the meaning of "to intuit".
Since we're quoting Douglas Adams in this thread, I'll mention something I posted a while back, with his writings as example. After Douglas Adams passed away, somebody was tasked to "finish" The Hitchhiker's Guide to the Galaxy :
> And Another Thing... is the sixth and final novel in The Hitchhiker's Guide to the Galaxy series. Written by Eoin Colfer with the blessing of Douglas Adams' widow Jane Belson
Being a rather big fan, I immediately bought and read this novel and I have to say that Eoin Colfer did a really really great job, nailing the tone, humour and writing style of Douglas Adams.
IMVHO, he did about as good as anyone could reasonably expect someone to do, when given this task. It was big shoes to fill, and I was impressed.
But it just also wasn't good enough, in a weird way that I found hard to put my finger on at first.
The thing is that Colfer was doing the tone of voice, even came up with somewhat new jokes perfectly in the style of, etc etc. And for the sake of argument let's say he was able to get "arbitrarily close".
But there was always one thing he couldn't do: Actually make something new happen, make a new kind of joke, do a real plot twist, a big reveal, stuff like that. Because then it would deviate from Douglas Adams' work too much.
However, if Douglas Adams was still alive, this limitation would not apply to him: he could make a new kind of joke, do a plot twist, big reveal, and it would become canon.
This the best "good faith" argument I can present for how LLMs lack "intuition", in some sense. Now "intuition" is not a very exactly defined term, but I'm arguing that the thing I'm describing here, is at least a part of intuition, that an LLM fundamentally can't reach (until they start getting their own volition, which I would prefer they didn't).
To address your question:
> Surely the AI can complete the prompt “Generate new research questions based on these observations”?
Yes I imagine it could do that very well. But it would still need a human to decide if the research questions are "relevant" or "within scope" of what the human wants (a.k.a. their volition). Without that filter, the research would just bloom out exponentially, with more and more questions nobody was asking.
And yes, up to some point that "blooming" behaviour is a useful aspect of research, the exploratory aspect/phase, but at some point you need to get back to the "synthesis" aspect/phase, to distill all the explorations back to "stuff that matters". And just like Douglas Adams vs Eoin Colfer, only the human who wants to know something, can decide to widen or change the domain of that synthesis, but if the LLM were to decide this (outside of exploratory phase), it would actually be considered the wrong answer.
And this is not at all to say you can't do research with LLMs, obviously you can. But this is just a thing they can't do, on a real philosophical level. I'm also not saying you can't work around this limitation, you probably can, I'm just saying it exists.
> People keep saying this. Why?
For the same reason that you can't draw a 15 of Diamonds from a regular card deck.
Of course you can. Tape a 7 and 8 of diamonds together and boom 15 of diamonds
People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps delivering more and more impressive results.
The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn, but I’ve noticed Fable to be quite a big step up there. How about politics? Will we develop new ways to let people express their own values in democracies, or will we just get much better at manipulation? How about experiment driven domains like biology?
We will get much better at manipulation and better at people “writing” things to justify their own feelings.
What’s new about LLMs is that you can scalably manipulate people individually. It used to be that you could either have scale (speeches, tweets, interviews, website, etc.) or individual engagement (replying to mail/tweets/town hall questions.)
Now you can pull the history and preferences of an individual, then shape a message—in real time—to them, specifically. You can have conversations on social media with a single person and shape your message specifically to them.
Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
Not new about LLMs. Targeted ads / big data is this.
Another degree of capability, yes. But we have been trending here for a long time.
I'm surprised no one has mentioned Cambridge Analytica.
What about them?
Cambridge Analytica gathered data to build targeted profiles and used these profiles for political advertising without informed consent: https://en.wikipedia.org/wiki/Facebook%E2%80%93Cambridge_Ana...
Yes, but there is little evidence this had a meaningful effect on votes.
Don't think that it matters if it was effective.
There's a reason attempted murder is a crime even if it was unsuccessful
It does matter though. If you want to murder someone by hitting them with a plushie, you're not going to get charged with attempted murder because it's not possible that that would ever work. There must be justification that the choice will have the intended effect.
We should not be gung-ho to give the government more power to regulate speech.
It isn't so clear there is a reason for the crime of 'attempt', or what or how good the reason is. The Star Chamber, which cooked it up, is universally condemned as oppressive. You might say they anticipated science fiction tyranny, when they invented the world's first thought-crime: they were able to make a crime of 'conspiracy' because they imagined that the speech of the conspirators was an 'outward act'; inevitably its glorious future was e.g. to jail the left wingers in the McCarthy period.
The point of view that says 'there is a reason we punish attempts' has difficulty explaining why we punish 'success' _even more_. There is a sort of bad concience about it. The paradoxes are discussed in a characteristically brilliant and twisted work 'The Punishment that Leaves Something to Chance' by David Lewis, one of the greatest philosophers of the 20th c. It nominally defends the law of attempt but can as well be read as a catastophically destructive parody of it. https://andrewmbailey.com/dkl/Punishment_Chance.pdf
It's interesting to claim that it would be difficult to explain why we punish successful crime more. A successful murder creates more suffering (victim's friends and family), and ostensibly the loss of a productive member of society. These all seem like fairly common justifications for punishing murderers under the banner of retributive justice.
I don't think the outcome is what makes things like this potentially illegal.
You're not allowed to bribe public officials. You're not going to escape the law by arguing that the bribe didn't get you the result you wanted.
Bribes are not speech though, and advertising is.
The way the documentary never questioned their sales guys claims felt naive
Personalized psyops against voters ...
so... advertising...
That's like saying a sling is the same as an assault rifle. Yes both are weapons but scale and capabilities matter.
It’s unfortunate that you’re being downvoted because this comment is true. And it just goes to show how far we’ve sunk that people have forgotten the problems of ad tech in the age of LLMs.
You’re making a weird number of assumptions about people’s voting and extrapolating to assertions about Society.
Ads are not really the same. They can’t be as tightly targeted to what resonates with someone. Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
That’s where this is headed and people are not ready. I don’t think we could prepare them, anyway.
I’m not making any assumptions.
> Ads are not really the same. They can’t be as tightly targeted to what resonates with someone.
The EU referendum in the UK proved your point false.
> Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
When people talk about “ad tech”, they’re not talking about TV ;)
And yes, people can and do engage with them. That’s how ads on social media works.
During the EU referendum, people were even resharing ads on Facebook without even realising they were ads.
> My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
I think that statement may itself highlight how prevalent manipulation is.
I fully anticipate all groups to continue maximal manipulation they can. One thing with LLMs is that it'll be a far less unified view, so a "divide and conquer" strategy is what I anticipate.
In US politics, the right is far, far better at winning elections than the left. This isn’t about personal preference. It’s objective political science.
Look at the most contentious issues in the US: abortion, climate change, taxing the wealthy, gun control, Affordable Healthcare Act.
The Democratic Party platform is aligned with national polling on every one. Every one of those issues has >60% support with voters and the Republican Party has blocked them all.
They play the game to win. And they do.
Objectively?
> Democratic Party platform is aligned with national polling on every one.
It depends on the pollster and where you're polling. I guarantee you rural Tennessee will not agree with downtown Washington DC on any of these issues. In contrast, rural California will likely agree with rural Tennessee. It's not as cut and dry as a homogeneous national poll of 2500 people. Every state, city, county are different. That's why there are federal, state, city, and county governments.
For instance, Abortions are legal nationally. States can individually decide how, or if, they wish to restrict it. This is as the constitution intends under the 10th amendment:
> powers not delegated to the federal government nor prohibited to the states are reserved to the states or the people.
This allows for democracy to take place at the local level, rather than having particular regions thousands of miles away from each other ultimately oppress the other.
To the point on LLMs, I think it's abundantly clear they will be used to propagandize and similar to social media will lock people in a bubble without alternative opinions. It'll be the worst of both worlds, the question is who's the puppet master. At some point soon, I imagine it'll be the AI.
So we need to "prohibit to the states" the power to enforce medical procedures (or the prohibition of a medical procedure) on individuals.
Which amendment protects someone's bodily autonomy or medical treatment from interference by the states?
If the Federal Congress passed a law providing that abortions are legal on demand up to the 22 week of gestation, after which it would require the corroboration of two doctors, would that stand, or would it be struck down by SCOTUS?
Same for legalization of marijuana, and many other "social" issues.
I don't know what distance has to do with it. I would probably be voted out of existence by my town, if the same constitution hadn't convinced them that they have no right. Independence Hall is remote in space and the events still more in time.
The pretense of Dobbs is that laws against abortion are like laws prohibiting medical marijuana or setting speed limits. This contradicts the purpose of laws against abortion and the actual language of most of them. It was obviously a desperate decision, irrespective of the truth about abortion
> Look at the most contentious issues in the US: abortion, climate change, taxing the wealthy, gun control, Affordable Healthcare Act.
You left out immigration, crime, and “moral values”: https://www.pewresearch.org/politics/2024/05/23/top-problems...
None of those are single issues, and the political fault lines are in the sub-issues. For example, while most Americans support "gun control," only 20% of Americans support a ban on handguns: https://news.gallup.com/poll/1645/guns.aspx. That figure has been trending steadily downward--from 38% in 1999 to 20% in 2024. That makes it much easier for Republicans to hold the line on that issue: portray all gun control efforts as a step towards confiscating handguns. That's hard for Democrats to defend against because most of the candidates, staffers, etc., who actually run the party probably are in that 20% who wants to ban handguns. That's simply logical, because handguns are used in the overwhelming majority of homicides committed with guns. It makes very little sense to have "gun control" without banning handguns.
The same thing for "taxing the wealthy." 59% of Americans think their own taxes are too high: https://news.gallup.com/poll/707951/americans-tax-views-rema.... And the difference isn't as big between parties as you think--49% of Democrats think their taxes are too high. So Democrats are in a position where they have to advocate for raising taxes on "the wealthy," without scaring any of their own voters into thinking that includes them.
Even a blind squirrel could find these nuts. Apart from that, republicans are ridiculously inept. For example, 2024 was the first time they spent real money trying to go after immigrant voters and minorities, and they made huge gains. But the on-the-ground operation disappeared after the election. Meanwhile, democrats are in these minority neighborhoods 365 days a year pushing their message.
> Apart from that, republicans are ridiculously inept.
They're not. They're really not. They're incredibly capable and are currently executing long term plans successfully, one after another. This goes all the way back to Regan and the disenfranchisement of education. They know exactly what they're doing to erode democracy.
Calling them inept is dangerously stupid at best, and at worst is just right wing propaganda to try and lull people into a false comfort and not act.
There's no "long term plans." It just seems that way because republicans are relatively ideologically homogenous, so it seems like there's much more top-down and long-term planning than there is.
Democrats, by contrast, have much more permanent political infrastructure. At the top law schools, for example, there's probably 10x as many liberal organizations as conservative ones. Democrats enjoy politics, so they are always out there organizing and engaging. Meanwhile, Republicans dislike the institutions through which politics is done, and so they gear up once every four years and then go back to their day jobs. Among the sort of more educated people who actually run the parties and serve as line staffers, republicans also self-select into private sector jobs, and out of public sector or political work.
Most importantly, putatively neutral institutions are de-facto aligned with democrats (because the professional class that runs these institutions is overwhelmingly democrats). Conservatives might have Fed Soc, but liberals own the ABA. Virtually every legal organization that isn’t expressly conservative or highly niche is de facto liberal and participates in support of democratic policies.
Someone conjectured that every organization that doesn't have opposing Leftism as its mission will eventually be run by Leftists.
It is way harder to manipulate people to do the right thing i think.
That presupposes that the left in the US wants to do the right thing. Something like government run grocery stores is not clearly correct and there is very little evidence supporting that it will work well yet it is a very popular leftist policy in New York.
City run grocery stores are certainly not evil.
Based on zero evidence. It’s very easy for a government to step in as a participant, ruin the profitability of a sector in an area, and offer a worse service.
They have no profit requirement or even revenue neutral requirement. So they can just operate poorly at a loss and still wreck other businesses because people will put up with breadlines to get bread for ultra cheap.
The general thing to watch out for with all of these “surely it can’t be evil to do nice thing X” is suicidal empathy. It can seem correct to your gut on the surface while it’s extremely destructive in the long term despite participants wanting to destroy something as an explicit goal.
... not overtly, intentionally evil.
FTFY
What’s the case that they’re secretly, unintentionally evil?
Bad economic policy is subtly "evil", by way of allocating finite resources inefficiently. Usually this is unintentionally done by not appropriately taking second, third, ..., nth order effects into account.
... which would make (our current model of) capitalism especially evil. Compared to a supermarket.
They destroy grocery stores that offer variety in every neighborhood they operate in. They offer worse service and people put up with it because the rest of New York is subsidizing them through taxes.
The margins for groceries are objectively thin. The only way to provide food at lower prices is to provide a worse good or service, eg. less variety, less quality, less availability, etc. You will see all of these outcomes in NYC, if anyone even accepts the bids to begin with.
It's harder to convince people to all agree on the same, different from the norm, thing.
If you've got 5 people in a car and you play ABBA in every road trip, then one day you suggest to change, the problem is not in being okay with "something else", but on agreeing what that other thing should be, set against the already known thing.
That's why there's so much infighting in the left.
Biology would greatly benefit. We barely understand transcription and protein structure. And the straightforward systems that we know like insulin have complex post translational modifications. So while we have a map of the partial proteonome, we have barely scratched the surface on networks regulation and interactions.
> And the straightforward systems that we know like insulin have complex post translational modifications.
insulin is not straightforward, the way the insulin molecule interacts with its receptor is nuts. on the other hand its post translational modifications are simple and dont have anything particularly surprising (no glycoslation, disulfide bonds where you would expect, nothing special kex2 cuts, arent really defective in disease states even)
I think the question is less about where it's most beneficial but which knowledge structure lends itself to LLMs most. Since biology's "language" is way more complex and irregular than maths or natural language, it isn't particularly accessible.
> Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
Side note: it's manipulation either ways because you chose what to talk about, with a goal in mind.
Actually, no. "Manipulation" is a negatively loaded word, and you wouldn't use that word if f.ex. someone helpfully & truthfully helps others see they've misunderstood sth.
I believe that the decentralization of manipulation (taken in a very broad sense as an effort to modify people's views) spells trouble for democracies. The mass media of old, with all its flaws, created a shared pool of information managed by well-educated people, some of whom understood that they had the power to keep democracy running (keeping populists away from the "manipulation machine," curbing blatant manipulation attempts, and so on).
Decentralized manipulation, by contrast, just runs amok creating echochambers and polarization.
the fact that the useless left/right divide is still so widely used shows that manipulation is working well even pre LLMs...
when it comes to the important question, then both "sides" are the same team.
or if you want it with a pinch of humor:
when a boot is on your face, it makes precious little difference whether it's the left or the right boot.
(i lived the first 10 years of my life in communism)
If you think that the American Left is anything like communism, I’m sorry, but, respectfully, I can’t take your comments seriously.
The left in the US would be center-right in Europe, who are certainly not communist (they have separate parties that are communists!)
Even the socialist strain of the US has nothing to do with socialism scaremongering about Venezuela, etc.
The "left" as in the party, sure, not that much of an authoritarian leaning.
The "left" as in the pervasive group that crawled out of Tumblr, took hold of Twitter back in the day, and has a stronghold on Reddit now? Those do care about what you can say, think, watch and read, and the more they can control, the better. The US right can only dream to have half as much control as the left has had in the last three decades.
The left has cultural power, the right has political power.
This take makes me profoundly sad. Too much culture war and to little empathy.
Excellent job not understanding the point. The party thanks you for your service.
Not at all. Economically, yes. Socially they are far left of the European left.
The parent comment wasn’t making that comparison. In fact, quite the opposite. The communism comment emphasized personal experience with a boot in one’s face, not that the boot was communist.
What even is the left though in the US? I haven’t watched any TV in decades, and the online content is entirely algo driven, so I don’t know what is happening.
both democrats and republicans take their money from the same donors. its all a show.
It's a show until you're rounded up by ICE or you can't get an abortion in your state (or others because of mass surveillance).
its a bit like how big companies will pullin a CEO who;s job is to do all the unpopular stuff and then summarily dismiss him after (while keeping the policies he created in tact).
ideally we should be throwing our support behind democratic socialists like zorhan mamdani but old school moderate democrats are just as bad as trump. they are juts happy letting trump do the dirty work. At the end of the day they are both working for a certain foreign government and not the interests of the american people.
> If you think that the American Left is anything like communism
no, you completely missed my point. why would i talk about communism if in the same comment i'm saying that the left/right divide is a fake distraction... a red herring?
what i said is that what matters is the authoritarianism/freedom scale.
i just mentioned my past to emphasize that ideologies like communism are not to be identified by announcing themselves as 'communism'... but by recognizing their actual shape and actions. and i have a little first hand experience in that.
> nothing to do with socialism scaremongering
What's "scaremongering" to you is "life" to GP. Besides missing their point, you're also saying "you held socialism wrong, we can make it work, if only if it weren't for this pesky ... reality"... Sorry, you missed their comment, and I can't take yours seriously :)
> We will get much better at manipulation and better at people “writing” things to justify their own feelings.
Yes, ai has strong narcissistic traits. And so do the people that own them. And pay for them.
In response, people in general will become fat more capable of recognising the manipulation. And will become more paranoid.
This reminds me a lot of the proof by construction for the 4-color theorem. It was only enabled by the advancement of computers and dissatisfied many of the computer scientists and mathematicians since it was a "brute force" approach.
I wonder if AI will end up being similar. Certain theorems get proven by AI but others do not. We haven't reached the limits of this yet and I haven't found a good argument for where those limits will be (I do doubt that there are no limits).
My wife is working on her PhD in microbiology now, using OpenAI to implement her research ideas. Genetics is just too much data, and she eats through tokens like nobody's business. I thought I was careless with them, but she barely lasts a full day before exhausting her quota. There's definitely a lot of value there, but dealing with the data problem is a big obstacle in biology. I can only imagine what she could get done with more capacity, though...
What makes it so token hungry? Is she directly using the LLM for genome analysis rather then having it write the data analysis algos?
We so don't know what plan she is using.
Eg.a 20usd/m plan usually don't cut it for professional work.
Likely not using / managing context windows properly and then having it re-read data it’s already gone through
This absolutely terrifies me, surely one misread piece of data or a hallucination here or there and a little “oh sorry about that, I guessed at this portion of the data to save time” and the data used is useless?
Sigmoidal, not exponential. It would be insane to assume an exponential curve
this gets more nuanced because "the sigmoids won't save you": https://www.astralcodexten.com/p/the-sigmoids-wont-save-you
If the sigmoid is incorrect it's certainly more correct than the exponential.
> https://www.astralcodexten.com/p/the-sigmoids-wont-save-you
The conclusion of this article seems to be "you should give ai the benefit of the doubt against all reason". Barf
Isn’t the point more “it’s easy to fall into the trap to believe that predicting when the sigmoid is going to bend is possible and the right heuristic is to instead extrapolate locally”?
That aside, I’d question whether applying the Lindy effect in particular to something that’s not really a life expectancy but more a growth rate is credible… or perhaps a bit circular since it “assumes away” the ceiling.
Nobody in this thread is trying to predict when the sigmoid is going to bend. Perhaps they should
A lot of HN posters thought it was already bending at GPT4o/GPT4.5. Turns out, it kept accelerating (at agentic tasks).
I suppose it depends what you're expecting. From my perspective it bent around november 2025
Predicting when the sigmoid bends is difficult and predicting how long until it unbends is equally difficult.
The simpler assumption is that over enough time, the S functions stack together for long enough that working backwards from exponential is a better predictor of reality.
These stacked S curves have continually been true with most technology.
It hasn’t bent already? 2022-2024 certainly seemed far more exponential than 2024 to present.
I find this astounding. 2024 to present thread is can write a coherent 15 line function to ... what exactly?
No future for research mathematicians othet than as tastemakers / agenda setters?
The author of that post is a prominent Bay Area "rationalist," who have had a quasi-theistic relationship with the concept of all-powerful AIs for a couple decades now.
So are they right or wrong about the sigmoids?
All of the point of that article is that most people that think something is a sigmoid think it'll bend just as they are publishing their analysis. And the article says, don't do that, assume it'll be related to how long we've been on the "goes up" part.
Nothing in that article says it's not a sigmoid.
I dunno. They've been saying this for decades, including in a very well read Harry Potter fanfic, and I'd always dismissed them as kooks, but maybe they're right in the end.
You are confusing Scott Alexander with Eliezer Yudkowsky.
Scott Alexander is a follower of Yudkowski and comes out of that general LessWrong space.
This is the plural they, not the singular they.
Saying the same thing for decades, when things progress over those decades along the general trend line that you were worried it would, is decent evidence of a prescient prediction. Climate scientists also seemed pretty kooky in the 80s when they sketched out their trendlines, but now their concerns are all over the headlines as reality caught up, and the same is becoming true of the AI safety people.
Everything has limits, of course. But if you've not yet seen a deceleration, it's reasonable to expect that at the least you're in the middle of the sigmoid, not the top.
> But if you've not yet seen a deceleration
We have though. The velocity remains high but the acceleration is decreasing.
The improvement between gpt 2-4 where staggering. to 5, 5.6? Much less so.
The results improve of course, but the difference is no longer mind blowing.
Most of the improvement now has come from agentic harnessing, which is unrelated to the acceleration of the model but the tooling around the use of the model.
So we are seeing a deceleration and the speed is still high due to high prev acceleration but its not growing at the same rate as before and the edges are starting to show themselves
I literally don’t understand, and have a deep distrust of people who say we haven’t seen deceleration.
The diminishing returns are colossal. GPT 4 is more than 3 years old now, and while Sol is definitely better, it’s better at a tremendous cost. And 3 years of development. And it still fails in nearly all the same places as 4.0.
I’m not some “zero AI” person, I don’t think this industry will collapse into nothing and we will go back to a no LLM world.
But to look at this industry, which basically began in November 2022, almost 4 years and a trillion+ dollars later and say “oh yeah, definitely riding that exponential growth still” is some of the wildest shit I’ve ever seen, and it’s so common.
I'm the other way around. For doing real, useful work, these things were barely more than toys even just a year ago. Now they're very capable when used well, and still getting better at a fast pace.
in the absence of a stalling signal. it's better to assume exponential and work backwards than hope the next bottleneck is impossible.
Since when does realism become hope? (It's the other way around)
That's a false binary. The right binary is whether it's a convergent function or a divergent function. A sigmoid is convergent, which is not well substantiated and more farfetched than a divergent function, even if it's true that 2^x is too optimistic.
I mean, "sure, technically correct is the best kind of correct" - but where we sit on this right now? It certainly feels exponential.
A lot of this sort these sorts of posts are just "appeals to geometry" (aka "cope"). This is coming, it's coming hard. Now you need to decide what you want to do with your life in a world where your smarts aren't as special as they used to be.
This is hard (believe me, I know). But what one ought do is not eschew progress and cling to the delusion that things don't change, what one ought to do is try to see how they can leverage these tools for greater and greater accomplishments.
Shh! Don't upset the hype train
Maybe more like do not interrupt the hype train while it is derailing?
> The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn,
isn't it clearly split between verifiable not verifiable ? what is interesting about that question.
I do not think it is so clear.
Programming has verifiable and non-verifiable aspects. Competitive programming, passing tests, and performance can all be verified. But translating English requirements into actual software, software architecture, taste, or UI design cannot. And yet over the last couple years we’ve seen huge lifts in all of these areas, not just the verifiable ones.
Verifiable areas I think are clearly seeing the most improvement, or are the quickest to see improvement. But we are seeing lots of progress in non-verifiable areas as well.
How much of the non-verifiable progress is a function of labs purchasing expert data vs. the models improving with compute is maybe another interesting question, but fundamentally I don’t see spend on expert data as something that can’t grow if AI revenues keep growing as well. And as models get better taste they can also help filter and generate new synthetic data for their next versions to train on. The limits of this approach are not so clear.
That is because there is human annotated data there. Every session you or I used, then of course paid human feedback on repos (such as the recently famous example of meta forcing their employees to).
This is _much better_ data than 1/0 verification, it is as good as a gradient.
Automatically verifiable tasks improve faster since well, its automated.
> we’ve seen huge lifts in all of these areas, not just the verifiable ones.
most gains are still coming from data. isnt that supposed to 'run out' though?
Labs spend billions hiring experts to generate new data, and better models can better filter existing training data and generate new synthetic data. There’s no reason for that to run out, it’s just expensive.
You could view this as just continually patching a leaky ship. But it seems to work.
> Labs spend billions hiring experts to generate new data
I thought this is mostly RL data. In my previous comment i was referring to pertaining data.
I remember listening to Andrej Karpathy talk in a podcast about how synthetic data in particular is used to generate more data for pre-training. I see no reasons for that to have changed. I think it is likely a lot of the new data they are paying for contributes to pre-training as well.
I would also be very shocked if they weren't filtering or prioritising existing pre-training data as well, for example to do curriculum learning or to avoid data that degrades performance.
thats the whole point of turning to formal verification:
Imagine a hypothetical oracle, call her MyladyMath, imagine you can turn to MyladyMath, submit a correctly formed dossier of axioms and definitions, theorems with proofs, and then a newly putatively proven theorem T. MyladyMath will complain if your dossier is malformed, and point out where and why. If the dossier is not malformed it will eventually read in the claimed theorem T, evaluate its proof and then either point out a which step is erroneous and why, or ultimately accept the proof.
Instead of a large corpus of human authored text, this map from dossier/theorem -> accept / reject is a huge implicit array of bits, something fundamental, and this weird gigantic array of bits that effectively describe all accept/reject responses MyladyMath would return exactly. We would never run out of "corpus" when it comes to math, if humans had access to such an oracle.
And we do have access to this oracle, and possess compact algorithms that describe the accept / reject bits. One of them is called MetaMath, a minimalistic verifier, which keeps the concepts of prover and verifier separated, this choice results in concrete proof objects (a sequence of step label references).
The machines are going to comb through all possible paths of the next N steps, for progressively larger N, efficiently compress those results in the weights of an LLM and then use the gained experience as the intuition for guided "not-so-brute"-force proof search, using the prior iterations intuitions to grade the surprisal of the N+1't iteration of results, etc.
The money will not stop flowing in that direction: all power blocs, nation states, militaries, banks, ecommerce, ... depend on cryptography. And the machines will soon do more rigorous proof search grounded in more balanced and objective observations. There is no responsible disclosure mechanism for flawed hardness assumptions in cryptography. It's going to get rocky, and the common man will wonder why the gods have gone crazy, wonder why they don't just pull the plug out of the machines, but nobody will in a staring contest to see who dares keep the plug in the longest (and dominate global cybersecurity).
It's the end of the age of artisanal mathematics, it will now become industrialized mathematics.
I think you’re making a category error in your definition of politics here. Certainly technologies can favour winners and losers, but the struggle is an inherently human one.
Yes, the rate and nature of the results being produced is impressive, and Anthropic have recently invested in building out more in-house capability to do life sciences research (see e.g. [1]).
Overall I'm excited for this acceleration in discovery, even if it's causing disruption to existing research workflows. I wrote a bit about it recently [2], after seeing Levent Alpöge's counterexample to the Jacobean conjecture.
[1] https://www.cnbc.com/2026/06/30/anthropic-launches-ai-drug-d...
[2] https://scholarlyfutures.substack.com/p/frontier-models-tran...
Life sciences will get way more interesting once Demis Hassabis completes his simulated cell and we have more genomic foundation models
We definitely are not on an exponential. Don't say we are because this isn't up for debate. AI progress is logarithmic the million dollar question is 2x or 10x for linear improvement. The nearest qualitative shift would be very fast inference so people could start writing real software on top of LLMs. A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
> A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
I think you have completely misunderstood what OpenAI have accomplished here. Almost certainly no one cares about the specific concrete results achieved; they only care about (a) how difficult it would be for an intelligent human to achieve the same feat (ETA: the feat is the proof), which can be estimated by the amount of time the problem has remained open/a public conjecture, and (b) how general this artificial "intelligence" appears to be, which can be estimated by the diversity of topics where it was able to prove a difficult result.
It's as if I showed you a dog that I had taught to speak German fluently, and you remarked: "What point is a dog that can speak a language that less than 2% of the world speaks? Nothing to see here."
On the other hand, if provided the financial incentive would mathematicians have solved these problems? Its not hard to imagine a world where some hard problems were not selected by the sparse experts for whatever reason (lack of interest, whatever), which could have been solved if someone was throwing down millions for solutions.
This is what most people seem to forget, OpenAI has probably spent more money the last year than all of math research has during human existence.
nothing prevented us from defining cryptocurrencies, auto-rewarding new math in objective manners, grading theorem surprisal objectively, etc. before the advent of LLM's.
being lucky enough to receive the opportunity of hiding in some academic closet, poking your hand out begging for scraps, was a different, perfectly alternative path humans decided to take instead.
It's a bit late to start standing up for your rights when the robot overlords arrive.
Thats true, although once it stops being a party trick I expect the LLM proof mining to also be paid by someone else and be expensive as shit, the question same remains: who?
> It's as if I showed you a dog that I had taught to speak German fluently
And you’re thinking this is an accurate comparison?
It's more funny than the dog doing differential equations I'll tell you that. Specially when it gets mad.
What is the y-axis in this claim about logarithmic improvements? Any exponential curve can be trivially turned into a logarithmic and vice-versa, and the y-axis redefined as "progress", with no loss of accuracy.
One example that always bugs me is when people point to "exponential" or "sigmoidal" progress on benchmarks. Benchmarks are artificial constructions (saturation at 100% by definition) and benchmark scores should not be mapped to these words when talking about overall progress.
Example - progress on ARC-AGI-3 at the moment is exponential, steeper than 2^t and e^t. Does that mean AI is progressing "exponentially" in the colloquial sense? No, it doesn't support or refute that colloquialism.
Likewise with MMLU saturation. We can't go above 100% by construction. Therefore we have a "sigmoid". Gah.
The colloquialism is not helpful to begin with.
The reason writing is hard might be that post-training pulls style into a particular direction.
In other words, big labs are much more interested in making "AGI" than in making a good writer, especially as what qualifies as "good writing" is rather subjective. E.g. before AI use of metaphors and rhetorical devices were generally a sign of a good writing. Of course, not if you keep spamming the same rhetorical device - but a stateless AI can't know which one it is over-using.
I expect we can squeeze a lot more exponential out of LLMs because they’ve basically shown that human “consciousness,” insofar as it’s composed of knowledge and rules for synthesizing that knowledge, can be represented mathematically in a very high dimensional space. Why does this “just work?” Nobody really knows, but it clearly does.
However, I also expect this squeeze will come at an increasingly expensive price — not just because of inefficient token usage, but because of fundamental limitations of LLMs as a model.
LLMs are letting us brute force our way through a lot of reasoning, but it’s hard to believe that such a generic model of intelligence will take us to the next frontier. We’ll need some fundamentally new approaches at some point. Maybe those will make achieving the exponential more efficient or maybe they’ll unlock even higher degrees of possibility. Who knows?
The transformer is Turing complete. It might be a tarpit though? I don't know.
I think a nice example is using them for arithmetic. It's a specialized deterministic process, so it's extremely wasteful to do it that way.
But they're good at finding solutions to things we don't know how to specialize yet.
So, to use metaphor, maybe the transformer-based models are like the FPGA, and then when we figure out the patterns in that system — all the different kinds of specialized reasoning — we can extract it into an ASIC?
I think this is one approach to AI safety and interpretability that could work, but would require labs to slow down to figure out how to extract circuits/algorithms out of trained LLMs rather than deploying the opaque artifact.
> Why does this “just work?” Nobody really knows, but it clearly does.
We know language has to be learnable by every human, so it needs to be really independent of any specific brain development particularities. If it was not accessible to babies there would be no more language next generation.
>let people express their own values in democracies, or will we just get much better at manipulation?
Both are the same thing. Two sides of the same coin.
The distinction is between information flowing from people to power (elicitation), vs. it flowing from power to people (persuasion). These are not the same, even if they are closely related.
I guess I wasn't clear enough, _generally_ democracy = manufactured consent by manipulation of people by 'power'
Does AI make real opinion easier to hear, or fake opinion easier to spread? Even if you believe wholly in manufactured consent, how easy it is to manufacture matters.
> exponential
Many many things are only useful when expressed in the physical world, and that introduces lag.
we gonna go full Pandora’s Star where the AI discover laws of physics that are impossible for humans to understand and create the wormholes
Today's models depend on inference time compute to get these results. The inference time compute available on any claude subscription is not comparable to the ones used to get some of these results (yes, in this case, it is 2000 USD total as noam confirmed, but some previous results took more).
In general, you can think of the process as generating massive rollouts in generation N, and then compiling in the verifier/human feedback("gradient") signal into generation N+1. The time taken to make the rollout in generation N, and separately the time taken to get the same rollout in generation N+1, each grows constant in some tasks, linear in more, and exponential in some.
In the end, this becomes bottlenecked by time. Today, we can make statements like "I generated all these successful trajectories with 2 weeks of compute, in the next model it will be able to do it in 7 hours of compute", but very soon you'll find yourself making statements like "I generated.... with 8 months of compute, in the next model it can do it in 6 months", which isn't really enticing the same way you can _technically_ brute force passwords but it just needs prohibitive amounts of time and money. That is the "plateau". Note that, this point is quite far away. For example, at any point if we agree it plateaus, today's known hardware techniques such as fixed function accelerators give you a 10-100x timeline reduction immediately allowing for a few more cycles of improvement. This is not to mention future innovations, but of course none of that is helping with the benchmarks where the time needed is growing superlinearly.
In many math and coding benchmarks, we are still in the constant phase. These are the massive improvements we see every few months. I'm not making any prediction of what will plateau and what will not as it's not possible to make an informed prediction about these things IMO. But the observed fact is that some have already plateaud as in, they don't improve with reasonable inference time (likely superlinear growth).
> will we need mathematicians to translate
Let's take a sudoku analogy. The model is initially just doing the random value algorithm, but lets say you the human are watching it. You make one of the usual reductions and interject "hey you can stop trying 8 here because of ....". Over enough examples, you get to a point where the model is _forced_ to learn the logical pattern. Next generation, it will skip that number. After this, you can peak the distribution using simple 1/0 RL. Doing _pure_ 1/0 RL works decent, but its not frontier as its a very sparse signal.
For that lift, human (or even a better LLM, but if you're trying to improve a frontier LLM, there is by definition no better LLM) feedback becomes necessary. This is _why_ it is crucial that these models interface in natural language and is also why the labs are hiring AI tutors by the hundreds. The "better LLM" case is what Kimi etc are doing by "distilling"(bad term for this) claude.
> But the long term is completely bewildering if you believe any of these trends can continue at a similar pace for the next few years.
For math and coding, for now we are in the phase where the times are just ... constant, so there's little reason to think it will stop soon. We still need humans to expand the frontier. It just becomes a matter of if its worth the cost of compute for running this generalized The Algorithm or not.
Given how well chess players internalized _many_ (not all) of alphazero's emergent chess knowledge, I am confident we wont have too much trouble figuring out any new math LLMs come up with, which will let us keep expanding the frontier by giving the LLM the next "lift". Only when we reach the stage where the time growth become exponential will this stop, IMO.
> but I’ve noticed Fable to be quite a big step up there
what did you notice ?
I've noticed that out of all LLMs I've ever used that Fable is the MOST LLM; the text it produces is abomination. It's impressive how much I hate it. It is such an awful writer - it assumes the reader has zero context and therefore gives every single bit of context and detail - which is nice if you're writing a legal document I suppose. But it uses, niche, $10 words to describe every facet of everything it's discussing. I had it re-write some docs and I ended up rewriting 1k lines of of Fable torment nexus text to around 100. Because guess what, someone reading highly technical docs has a knowledge base that allows us to compress the topic into a much tighter representation.
Just praised Fable in another comment but what you're saying is also insanely true.
I literally roll my eyes and cringe quite often at its output pretty much daily.
I don't like to overload my sessions with skills but I've been using a "write-normal" skill I made just to have it rewrite outputs that particularly piss me off.
https://gist.github.com/alasano/1c734fa055231a5defcfd213217e...
I'm sure there's a million of these skills out there, but this one is tailored to the stuff that makes me mad in particular.
"the test is whether the sentence would work as a pull quote or a LinkedIn post. If it would, rewrite it until it would not." LinkedIn really became the default garbage example
Fable is much better at handling nuance. Opus/GPT 5.6 Sol are much more likely to miss the point you are trying to make, emphasise the wrong thing, exaggerate the importance of unimportant details, or introduce contradictions.
That said, Fable is still not a great writer, largely driven by it not knowing what it should exclude, and it still having the usual LLM-isms. But it’s better.
That's really what got everyone hooked in the first place.
5.6 Sol is great but there's a depth to the understanding that Fable exhibits that's unique to it currently.
Can I truly quantify this? I don't think so. Just that I spend a ton of time with various models and a certain point it's just a personal impression or a gut feeling.
In the days after Fable first came out I increased the amount of parallel planning of tasks that I was doing by 2-3x because it felt like I didn't need to be paranoid due to that handling of nuance.
Another interesting question is why the frontier labs are piling on pure maths, which has little direct economic value compared to something like law or improving the efficiency of their own models? How much OpenAI and Anthropic are paying to serve these models for ordinary users is the elephant in the room. A cynical take is that the frontier labs are trying their best to pump up their pre-IPO valuation through flashy headlines.
> Another interesting question is why the frontier labs are piling on pure maths
The reason is that the original scaling axes (parameters, training tokens, test-time compute) have saturated already, but RLVR (reinforcement learning from verifiable rewards) is still scaling well. And math has this nice property where you can synthetically generate arbitrary volumes of rewards to train the model, because math is self-contained and completely objective. Open-ended reasoning and analysis don't have that convenient property, and that is why progress is much slower outside of math and coding.
do we have any concrete idea of how well the models are scaling now? I agree with these 10 results being impressive, and it is easy to think "wow, and last year the models were barely able to solve IMOs problems". But for me it is perfectly possible that a non sofic group could be found by 100 good IMO students working on all the different strategies that have been proposed (OpenAIs solution was based on "expander graphs", which were introduced to solve the problem some years ago), so it could be that current AI is simply many (say 1000) old models working in parallel. This is linear scaling, not exponential. It could be I'm completely wrong also, the problem is that we have little information.
It's one of the few areas where you can verify results. That fits nicely into training models. They aren't just making judgement calls on what would be nice, it's "what can we do?".
If everyone publicly said that the models can only do things that humans have already done, but you know they can do more, wouldn’t you want to show them otherwise?
Math ability also helps with other things like making models more efficient.
Because it's a tool in search of a use case (or many use cases) and mathematics is the most natural use case for it. Mathematics is by definition the art of putting words on a page in a rigorously defined "correct manner" (i.e. in the form of a valid logical argument, a proof) and all LLMs do is put words on pages and evaluating if they're good words is by far easiest when there is a strict definition of right and wrong.
What's the best way to apply it to legal problems? Finding bugs in statutes? (there are often statutes with wording errors, missing negatives, things like that which don't get picked up for ages)
>Will we develop new ways to let people express their own values in democracies, or will we get much better at manipulation?
Yes.
>Will we develop new ways to let people express their own values in democracies, or will we just get much better at manipulation?
Is there even the tiniest reason to suspect that the people steering this progress will use it for the democratic good of all?
It's like Musks duplicitous argument about unlimited abundance. We have a lot of abundance now, we just keep accelerating it all into the hands of fewer and fewer people - whose response is only to want more, and more, and more.
Yeah, we really should just storm the facility where Musk is hoarding all the worlds bread and meat.
Wealth in terms of capital doesn't represent material goods, it represents the system's confidence in your ability to direct capital efficiently. But eventually efficient capital bottoms out at consumable goods. Someone like Musk with a lot of capital under his control is contributing to the end goal of unlimited abundance.
With his shitty cars, orbital garbage, or CSAM generator? Or his moneyed attacks on democracy and communications? Or his vandalism of government programs without insight, experience, or qualification?
In a free and fair market, his capital would be regarded as a deeply inefficient distortion.
In fact the opposite, look at talks by Peter Thiel. At least he’s being honest. These people are bastards, but for some reason moral goodness and wealth has been conflated leading us to venerate greed.
There are many math problems that are simply puzzles: intellectually interesting but nothing worth of value depends on it. To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that.
The results OpenAI demonstrated are impressive, but it also looks like they threw a lot of compute at it just to get results. How many tokens did they waste on problems they couldn't solve? Applying inference infrastructure on a large number of math problems at scale we haven't seen before to me doesn't demonstrate an exponential curve in model abilities.
"To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that."
I was recently listening to BBC Radio 4's episode on the Poincare Conjecture[1] and the guests on the program were discussing how the problem that looked deceptively simple eluded the great mathematicians of the time (including Poincare himself) for nearly a century and how Grigori Perelman cleverly came up with the proof. It took other mathematicians working in groups years after Perelman's publication to understand and validate his proof. The mathematicians on the program were speaking of highly of his proofs and admiring the originality of his work. This made me think of one neat experiment where if we cut-off a frontier model's training data 2002 or anytime before Perelman posted his proofs on arXiv and check if it can come up with the solution by itself. That would surely be a great signal to see if these LLMs aren't just solving interesting puzzles and that they can came up with something truly novel.
P.S I highly recommend Misha Green's "Perfect Rigor" for anyone interested in the history of the problem and the genius behind the proofs of the conjecture - Perleman. I found it an entertaining read and could digest its description of the problem as a layperson (with undergrad level math).
[1] https://www.bbc.co.uk/programmes/p0038x8l
Pretty cool. The impact of AI is getting undeniable, there aren’t many positions left to move the goalposts to at this stage, next they’ll have to be outside the stadium entirely.
The sooner people can be broken out of their denial about all this the better, and we can start actually taking it seriously.
The models are frequently getting worse at items that they aren’t being benchmarked for — and that’s happening more and more over time! Other people in other fields aren’t idiots, they are accurately perceiving the fact that these models are being hyper optimized for our industry, and are becoming less capable in other domains over time. Models of the same scale are massively worse at writing a broad variety of styles of prose than their equivalent from two years ago. (Models of increased scale are a mixed bag.)
Maybe you’re the one who needs breaking out of your cached beliefs.
Proof?
In my experience modern models are better at all tasks than models from two years ago, especially complex multi-step tasks.
you are working on coding. they are working on things like "creative writing" remember that gpt 4o was popular among those who had ai as a romantic partnet?
> remember that gpt 4o was popular among those who had ai as a romantic partner
I suspect GPT 5.6 would be even better at it, if given the same sycophantic system prompt and lack of guardrails.
Dude it's not a system prompt, it's the training.
The API lets you adjust the temperature. Lower values introduce more deterministic outputs, which likely helps with the hallucination rates.
If you want creative writings, use the API and play with the sliders.
They actually removed the temperature parameter starting with GPT-5.
I highly doubt that.
sycophancy
It wasn't "better" it was better at kissing your ass which matches what a lot of people want in a partner.
gpt4o & associated parasociality is considered an alignment failure and is actively trained out of the model, so that is a terrible example of regression
Well that's on purpose lol. OpenAI does not want you falling in love with their chatbot and have been deliberately training it to be less romantic.
There have been several cases of suicide and self-harm related to 4o, AI psychosis is a real risk and will probably be in the DSM
sure they do if it makes them money, probably just not worth the controversy right now
For customer support I don't think models have gotten better since gpt-4.1. The class of small models, with limited to no reasoning, that need to handle a complex issue with a touch of empathy, has not improved much.
I think most are actually worth, as agentic harnesses seem to optimize for solving poorly described problems rather than following complex procedures as written. In other words, instruction following maximizing models seem to make worse free-form agents, but they're really all that some domains need.
I understand the point (I don't agree with it; tool calling has gotten much better/reliable and that is very important for customer support) but consider: If you can get same for a lot less, that's an improvement. If we found a way to supply fresh water and electricity for -90% cost after 2 years, that would be fantastic.
You can do many more things, when stuff is cheaper, even if the stuff were otherwise unchanged.
>The class of small models, with limited to no reasoning
What? GPT-4.1 was not a small model! And why wouldn't you use reasoning?
You're of course going to see poor results when you restrict yourself to small non-reasoning models, but why would you?
"Small" was a poor choice of words here, "low compute budget" is more what I'm getting at.
In voice interactions, ttfat is actually relatively important. If you look at models with a <1s ttfat you eliminate almost every reasoning model, less some of the diffusion models and more obscure ddtree/dflash like speculative decoding implementations.
https://openai.com/index/introducing-gpt-live/
GPT-Live, which is coming to the API soon, responds instantly while reasoning in the background. So it can say "Hold on, I'll look that up for you" and continue to respond to the user conversationally while running an asynchronous reasoning task in the background.
It's not in the API yet, but it should be in the coming weeks. You'll see an enormous improvement compared to GPT-4.1.
Do any of the big AI companies have a model that are good at tasks that require learning?
For example, every day people teach teenagers how to drive and with only dozens of hours of practice, they are on the road.
is this not essentially what ARC-AGI-3 is? i agree that in-context/continual learning is somewhere the models are still mostly weak at
I don't think any of the ARC-AGI-3 tests are very interesting. At least not as interesting as driving a car. Children literally do a similar task in go karts every day.
Another interesting task would be to take the AI in a robot body into a vegetable garden and teach it to pull weeds. This is another task that lots of children help out with.
Data 'compression' collapse. People publish AI generated slop on the internet -> next generation of AI is trained on that data -> the lossy/fuzzy training make the output worse -> rinse and repeat.
So your answer is: ignore the progress, it’s not really happening, actually it’s getting worse.
That’s not a credible position, but there isn’t anything that I or anyone else can say to someone who simply doesn’t want to believe something.
It sounds like they are making a clear argument: models are getting worse for certain domains even while they are getting better at others.
I don't know if I agree with that but it doesn't seem like an irrational claim and does seem credible to me.
I just don’t think it’s true, or is significant enough to matter to the direction of travel of AI even if there were something to it. It’s another cope post being lobbed at the idea of AI going somewhere and I’m sick of them.
"The sooner you can be broken out of your denial about all this the better, and we can start actually taking you seriously."
every lab independently discovered that getting good at bit alchemy (coding and related tasks) should come first as it will enable the formation of training pipelines that will then solve everything else.
so far there is no end to this progress in sight so it's full steam ahead on this singular domain. once it plateaus you should expect to see the greatest disruptions in human endeavors ever as all the training flops will start flowing to other domains to disrupt and dominate.
Never understood all this talk about moving goalposts - you understand that's how science works, right? We improve, we learn, we recalibrate our expectations based on what we've learned. If we never "moved the goalposts", we'd be stuck scoring the same goals over and over.
> We improve, we learn, we recalibrate our expectations based on what we've learned.
That's not what people mean when they say "moving the goalposts". It means that people are adamant that something wasn't important/hard/impressive once the "AI" solves it. And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
It's also very much not a new phenomenon. It's been happening since the 1980s. As you can see from this quote from GEB by Hofstadter:
> There is a related "Theorem" about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was first proposed to me by Larry Tesler, so I call it Tesler's Theorem: "AI is whatever hasn't been done yet."
Yes, this is exactly what is meant by “moving the goalposts”. And it’s a fairly well known expression applying wherever people retroactively change their requirements in reaction to those requirements having been met.
It's almost like I disagree with your use of the phrase in this context, rather than that I don't know the meaning of it.
So you defined your own idiosyncratic version of "moving the goalposts" and used it to rebut his argument, with a condescending "you understand that's how science works, right?"--instead of being honest that it is you who are changing the definition, and not his failure to understand anything.
I don't see how that's any better.
He rebutted the argument about moving the goalposts by moving the goalposts. It’s better by dint of sheer bravado.
You got me - I've been rumbled.
Actually, I'm really an LLM and this whole thing was a clever bluff.
How were any of us meant to know that you were using your own personal and undisclosed definition of a well-established phrase?
“AGI will only, truly, be achieved when the machine can destroy an industrial-type toilet after downing a Supreme Burrito and a large Baja Blast.”
-Alan Turing (allegedly)
> And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
The fundamental argument that I've personally made since the early days of this is that LLMs are not reasoning, in the way that word is commonly understood.
There are lots of reasons why that argument needs to evolve that could certainly appear to be "moving the goalposts", but let's take an example.
A lot of AIs were tripped up by the question "Should I walk or drive 50m to the carwash?" Several folks liked to use that as an example that illustrates that LLMs aren't reasoning, but as the models have been trained on that specific example, it's of course less useful. An AI can mostly nail it now.
So a different example is needed. A new demonstration of how these things fail at basic reasoning a child can do.
Did I move the goalposts? I don't think so. The fundamental argument stays the same. It's not hard to find lots of examples that trip up LLMs, because they are what they are: statistical inference machines. Nothing more and nothing less.
Useful, sure. But also commonly misapplied to areas for which they are inappropriate solutions.
If it seems like I don't understand the meaning of that very well-known phrase, then clearly I have failed to make my point. I'll try again. And please note that I will use some generalizations to make my point more clearly, rather than because I don't understand nuance; kindly grant me a charitable reading.
In recent years, I have commonly seen the phrase "you're moving the goalposts" deployed by the "it might be sentient" crowd to shoot down the "it's a stochastic parrot" crowd when the latter respond to a new development with "OK but...". In a well-understood field of inquiry, that would be a clear case of goalpost-moving, in the commonly-understood meaning of the phrase where requirements are retroactively changed in response to them having been met. Thank you OP. 'Artificial Intelligence', and indeed intelligence in general, is very much not a well-understood field of inquiry - in fact we don't even have a common agreement about what 'intelligence' is. We are therefore learning as we go (even after all this time!) but making rapid progress in recent years. When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change? This is arguably one of the most pathological development projects ever - what are the requirements? 'It thinks like a human'? What does that mean? And the answer is we don't know what that means, and we're working it out as we go - moving the goalposts. If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
Side note that, in case it's not obvious, none of this detracts from how impressive LLMs are. They're a marvel of the modern age, all the problems notwithstanding. However I reserve the right to stay sceptical about their capabilities.
> When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change?
It's in how they change, not the fact that they change. The skeptics seem to have secret definitions for intelligence, sentience, consciousness, creativity, etc. that amounts to "a thing only humans have". Often that thing is equivalent to a soul. When yesterday's challenge (LLMs don't have X because they can't do Y!) is met, Y changes but X stays the same. This is not the process by which a field matures, it is a rhetorical technique used by skeptics to avoid honestly stating or confronting their internal definitions. That can be revealed by asking the skeptic the following:
"Forget LLMs. What if we made a completely physically accurate simulation of a human being?"
Many say no, that simulated human being still couldn't have (intelligence, consciousness, sentience, creativity, ...). This reveals that there is a necessary metaphysical component to those attributes, at which point any scientific-minded person will leave the debate.
OK, I see, thanks. That applies to some skeptics, though - I'm in the AI skeptic camp but only in the sense that I believe we should make sober assessments of what these things are actually capable of.
Thanks for this clarification of your position. The brief answer is:
> If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
The criticisms are directed toward people who did clearly act like they knew, not the ones who were honest that they did not know.
> If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
To me, the goalposts were already defined by the person you were responding to. "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
Note that this does not depend on things like AI sentience or defining "intelligence" more rigorously, it just depends on AI impact.
Take your point, although
> "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
those are not goalposts - that's an incredible vague 'goal'. What are we moving here exactly?
The "goal" here, not in the sense of what AI scientists are shooting for, but in terms of "goal", "goalpost", and the idiom of "moving the goalposts", would be to invalidate the current argument. Typically, you "goal" when you successfully disprove the other person's argument.
Thus "moving the goalposts" is for the defenders to shift what argument that the other side is trying to disprove.
If the argument is "the impact of AI is getting undeniable", you'd try to show that impact of AI is not that high, and not looking to be that high in the near future.
The other side might move the goalposts by saying something like "well, it doesn't matter if AI is actually that broadly impactful right now, all that matters is whether it's impactful in the field of mathematics". At which point, you might be frustrated because the effort you put into engaging their previous argument is now moot.
PS - yeah, yeah, I overexplained
What you are doing is "motte and bailey".
The motte is "AI useful". The bailey is "Singularity is nigh".
The unified position which many folks deny is "AI is powerful" or, alternatively, "AI will be powerful soon".
(I'm personally still skeptical about this, but I'm being pulled towards accepting it).
"AI is useful" is too low of a bar, and "singularity is nigh" is too high. "AI is on its way to upending society" is about in the middle, and still vastly contentious among laypeople.
>> The motte is "AI useful". The bailey is "Singularity is nigh".
But there are people like Ed Zitron, frequently posted and cited here, who disagree even with the former.
I think Ed Zitron is mentioned just because his name is Zitron. He just repeats ad nauseam opinions concentrated around one simple, very boring pole on a wild and interesting landscape of emerging reality. Anybody could be doing that. A lot of people do that. Yet no other is named Zitron. And that's why I heard name Ed Zitron hundred times. That's how you become a voice of (a part of) the generation. Just have a memorable name and repeat the same opinion over and over that people can flock around comfortably. Content is irrelevant.
Personally I prefer to follow explorers rather than swamp-sitters.
to an extent zitron is saying its not useful, but as a more nuanced opinion, "ai is not cost effective, nor is it improving profits or revenue"
From a random article I grabbed of his (https://www.wheresyoured.at/subprimeai/):
"it isn't clear whether generative AI actually provides much business value at all"
"cannot seem to find a product that people will pay for, in part because the results are so mediocre"
"Last week, we got our first real, definitive glimpse of what’s around that corner that future. And boy, was it underwhelming."
"OpenAI claims that o1 “performs similarly to PhD students on challenging benchmark tasks in physics, chemistry, and biology.” Just not in geography, it seems. Or basic elementary-level English language tests. Or math. Or programming. "
"Worse still, it's kind of hard to explain why anybody should give a shit about o1."
"o1 shows that OpenAI is both desperate and out of ideas."
"the software is not becoming more useful"
Honestly, every other line is quotable in this context.
Yep, he is a PR stunt guy, and the number of videos that come up when you type Ed Zitron into YouTube should tell you how many people are eager to feed their cognitive biases.
It’s comical and honestly incredibly embarrassing…
But it seems we have somehow optimized away shame. It wasn’t good for profits, I guess.
They aren't claiming that science doesn't progress by moving goal posts. They're talking about how critics of AI have claimed it isn't revolutionary/useful, then progressively changed what would it mean for AI to be actually revolutionary/useful.
Not long ago many folks were saying AI was the same as the crypto bubble. No real useful technology and only hype.
Did you know that "revolutationary" is not equivalent to "useful", and "revolutionary" is quite ambiguous?
I don't know if you're trying to dunk on me. I didn't intend to imply they are synonyms.
I think AI is clearly both revolutionary and useful. Revolutionary insofar as the job I do has changed almost completely in a year or so span.
> Never understood all this talk about moving goalposts - you understand that's how science works, right?
I agree with the parent that we need to acknowledge that we're at a turning point in history. I lived through some of them (internet, ubiquitous personal computing). But it's somewhat difficult to comprehend the impact of this one for many people.
I do biomedical research at one of the top European research institutions. We're very well-funded, but I can clearly see the gap between us (say, top-100) and top-10. I also realize this gap is going to get so much wider unless we invest heavily in AI access (and I'm not so sure I can sell anything more expensive than $20 Claude subscription to the leadership).
I think people having 6-7 figure SOTA AI budgets will move exponentially faster than those who don't. That makes me worried.
So, for me, it's not a question of recalibrating expectations. We're way past that.
> The sooner people can be broken out of their denial
There is irony here
People will be broken out of their denial by actual economic growth. That's what this is all meant to be for... I think we might start seeing some surprising numbers.
How do I directly benefit from this supposed "growth"? Seems like its just making things worse for most people.
Plenty of people are making money from AI? And I am not sure I've seen it "making things worse for most people" yet, do you have a source behind this?
If your source is AI layoffs, there were plenty of layoffs with the invention of the horseless carriage, but that doesn't mean it made humans worse off overall.
There are reasons to be skeptical about progress and AI and all that but the 'making things worse for most people' thing you mentioned seems yet to be based in any reality
If your answer to "How does AI benefit me?" is "Well a lot of people made money" then I'm afraid you've lost the plot.
most devs i talk to say that they would love to go back to coding without ai, if everyone had to do it, so there's that. Oh also scammers have gotten so much better with llms, so a significant amount of peoples lives are worse due to that
>Plenty of people are making money from AI?
Is the amount of people making money from AI greater than the amount of people who lose money from AI? Is maximizing prosperity across humanity even a goal, at this point? Am I to be expected to believe, without doubting, that the end goal is the benefit of the many?
Would something like "services deflation" even show up in the numbers? That's what I'd expect to happen first
deflation will just be inflated away, always
But who benefits from that economic growth? If it's only the capital owning class, and the rest of us are left in the dust, then the growth is irrelevant.
I ,for one, have read enough history to know that it's never the proles who end up benefiting.
I suspect that you have not read enough them. Ask someone in China (which yes, still has a capita owning class) if the proles were better off 50 years ago or now. The only real probes we have now in the west is that we have failed at housing, precisely because instead of growth, we have made many choices to increase returns for old people, and insufficient taxes for real estate.
I was a bit triggered by your China comment, if only given the fact that there is a very specific institution that drove many formerly prosperous cities and communities into abject poverty (the CCP in Mao's era) and then tries to claim all the credit for the market driving how many people they've lifted out of poverty.
And nowadays with Hong Kong (and probably soon Taiwan) they are proving they are perfectly happy to destroy economic growth as long as it benefits The Party
It's indeed very exciting. I'm looking forward to new advancements/predictions in physics. Preferably as beautifully explained as E = M*c^2
A formula like e^(i*pi)+1=0 is probably closer.
But it's not clear if LLMs will produce abstractions that humans would find elegant.
Can someone explain why we’re not using it to solve the obvious big conjectures/problems though? If this stuff is solved, why isn’t Riemann the first thing to go? Even if it means a fund of several hundred $k to let it churn on it.
https://garymarcus.substack.com/p/openais-amazing-but-vastly...
https://garymarcus.substack.com/p/two-critical-updates-re-as...
As always, PR hype. Goalposts have not moved.
Guys, please use critical thinking. The haters don't hate by default, we hate because we're gaslit about this stuff every day and it's annoying. Extraordinary claims require proof, and they're not giving us information that would be essential to knowing if this is actually significant or not.
AI already have an impact, and yes this is PR hype because this is a product. Yet both can be true at the same time. We're not blindly eating what's OpenAI is serving us as gold truth, we're just admitting it's doing remarkable progress.
Remember October 2024 Pelicans [1] ? It's been only less than 2 years.
We don't know what will come in the next 2 years. But the progress doesn't seem to stop for now.
[1] https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
The pelicans are still not ok to this day.
People are skeptical of the announcement because the room include several PHDs in math and physics. The prompts are not published so we can see how generic the starting prompt is.
> The pelicans are still not ok to this day.
If I could take one out of context quote from this whole thread as a response to TFA, it would be this one.
Nobody is saying it doesn't have an impact. What we're saying is the near-religious fervor isn't warranted. AI boosters always speak in the future tense, which is extremely telling because we have right now is basically "ok" not earth shattering.
If you think it's "ok", then you're probably not using it the right way.
I've been analyzing thousands of x86 traces and the AI made sense of them at a scale no previous human or algorithm could do before - I've been reading assembly everyday for half of my life. I'm doing prototypes in minutes what would take hours or days before. AI is answering questions in such a precise way in so many domains, learning something has never been easier.
This is "just ok" for you ? What do you need exactly ? god omniscience ? come on, can't we just appreciate for what it is and be excited about the future ?
Gary doesn't argue it's hype though. He argues 2 things: other people are getting carried away with the result, and we don't know enough about how it was reached to know where it falls on the impressive scale.
He literally says it's an impressive feat in the second article.
> other people are getting carried away with the result
That is the intended purpose of hype.
He's polite, but he's basically saying there's potentially a lot of smoke and mirrors. I agree.
best point in that first link: openai doesn't tell us if they only tried to solve these 10 and each was solved (amazing) or if they asked it to solve a million problems and it got these 10. Either is great, one is more so.
Gary Marcus has been moving the goalposts since day 1. The guy is a psychologist. Why would anyone care what a psychologist has to say about AI? He's likely made good money from constantly moving the goalposts and being a denier, due to the publicity he gets.
Because he's not talking about AI, he's talking about people's psychological reactions to AI
That's just not true, at all.
He's simply a good phone number to have for journalists under time pressure who need to add the contrarian voice to their upcoming story. He delivers it reliably, then never reflects on how he was wrong in the past, just blasts forward as if nothing happened and just makes the next bonkers claims to the journalists who are very thankful for the prompt delivery of how AI is a nothingburger, and fake and won't ever do XYZ that it then proceeds to do in N months.
I remember the time when he insisted that diffusion-based image generators trained on Internet scale data will never be able to make an image of a horse riding an astronaut. Today you can generate 4K video of that.
And yet Sora folded because it's way too expensive to run a service like that.
A lot of his predictions have held up very well.
Mate, if you don't like Gary Marcus that's your call, but you're completely just lying when you talk about his qualifications. He is not "just" a psychologist, he's done a lot of AI research and even started AI companies that were acquired. This is absolutely a person with the credentials to speak on this subject, and honestly all his takes I've read have probably been TOO nice to AI companies.
Your comment seems entirely disconnected from the posts you linked. It's impossible to deny the results, it is not PR hype that in the past couple of weeks LLMs have resolved problems that have been open in mathematics for many years. Some of those problems had remained unresolved despite keen interest from many humans.
The only way that is PR hype is if you're invoking the insane conspiracy that frontier AI labs are just buying off results that would otherwise be career defining for a mathematician, just for marketing.
The posts you linked are urging caution regarding the exaggerated e/acc-esque lies peddled by people like Musk, not that the models haven't proven themselves as having genuine ability to contribute to research in some areas.
Society at large is getting worse at critical thinking, because we are increasingly offloading that thinking to AI
I use these strange machines all the time. They have gotten notably smarter. That’s my personal experience.
They still do things that I find incredibly annoying and “dumb”. And I still have to clean up messes they make quite often.
But on the whole they are clearly smarter than before. No extraordinary claims needed. I just try to learn how the tool works and how to use it effectively.
Those that claim their opponents are in denial are most likely in denial themselves
How's that going for climate change and its denialists?
it is very hard for people to eat crow, as the replies will show
Really? has anyone ever claimed that ai will never be able to prove theorems and conjectures ?
yes, in fact I recall many people on this exact forum saying that AI will not be able to novel work at all.
i can link you likely dozens of comments from people wrong about this replying to me over the last 5 years
Hundreds of comments with 100% confidence AI/LLMs only regurgitate existing knowledge and can not produce anything novel
And what does taking it seriously entail?
In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
How does intellectual property work in an AI generated future? What about healthcare advances, who gets to own those?
What about meaning, what about purpose? How do we replace the work ethic that tells us we are our jobs and idleness is immoral? How do you replace “What do you do?” As one of the first questions you ask a new person?
That sort of thing.
I think this too is a kind of denial. As in, while some are in denial about the usefulness of AI, others are in denial about whose living standards are actually going to be uplifted.
And it's sad, really, because I think these two groups would make a great pairing if they could stop arguing against one another for a moment. They'll both be impacted about as much and probably have the same ultimate goals (to lead dignified lives).
But it seems these days everyone is more interested in Kayfabe and feeling like they're in the right than working together, so maybe I should just keep quiet rather than attract the ire of both groups...
> others are in denial about whose living standards are actually going to be uplifted.
I don't know if it is fair to say they're in denial. For my part, I don't expect life to get much better for regular people (especially short term), but that doesn't mean we shouldn't work to try to make it happen.
Asking questions about policy and values and pushing to have those resolved in positive ways is about as far away from denial as you can get. It’s possibly the only useful thing an ordinary person can do.
What a lot of people want to do, and I’m not saying that you’re one of them, is to assume that a positive outcome is impossible and either do nothing or loudly yell that the world is ending. Neither is particularly useful.
Or, as I said above, others just deny that there’s anything to see here and try to get people to move along.
The AI companies themselves, who are highly motivated to sell AI as overall positive for society, notwithstanding some security/etc risks, and who have economists on the payroll to think about things like this, do not seem to have found any possible positive outcome to present.
Shane Legg (DeepMind co-founder), one of the more intelligent and thoughtful people you'll find in the industry, could only offer "it's a tough problem - we need to think about it" when recently interviewed by Hannah Fry.
On the surface the most likely outcome for AI allowed to replace jobs is extraordinarily negative, especially since it is a general capability technology, not a specific one where displaced workers can just move to another field. Once AI becomes more capable it will be able to do the vast majority of white collar jobs, including any new ones that may appear as a result of AI. As Shane Legg put it, "if your job can be done remotely, sitting in front of a computer, then it can probably be replaced by AI".
Not only does AI threaten to replace ALL the white collar jobs, but it is rapidly going after blue collar (factory jobs, driving jobs) and pink collar ones (Japanese robotics for elder-care) as well.
If a positive outcome (which doesn't include putting displaced workers on welfare - UBI) is possible, then it sure would be nice to hear it, and the silence from the AI companies, and government for that matter, is deafening.
UBI probably is the positive outcome, though it may not seem like it to begin with. Initially it will likely be stigmatised and under-resourced, but as a larger proportion of people move out of work and onto UBI that stigma will drop and the resources should grow.
Eventually UBI will be the norm, and if the living standards of a person on UBI is as good as yours or mine today, that will be an enormous win for everyone. It’s like pensions, once these were only for the elderly poor, now they’re a right for everyone in most developed countries.
It’s also interesting that for most of human history leisure time was the point of life, and only in recent modernity has work come to be the meaning of someone’s existence.
UBI has to be commensurate with production being automated. That’s a big logistical problem, if you think building datacenters is a challenge try bringing about radical abundance, but even so it’s not insurmountable. It just needs to be taken on as project and not seen as an impossibility.
So much of this is not about what is possible so much as what people believe is possible. We can do anything if we try.
We don't see a progress towards UBI even inside individual countries and with AI we will need it on international. What people in Europe or Africa will do when all jobs there will be replaced by American or Chinese AI companies, where they will get money for UBI?
> Eventually UBI will be the norm
I see comments like this tossed around a lot, but what makes you say this? Don't you think its more likely that most people end up in poverty?
No, I think people will demand to not live in poverty. Capitalism allows people to escape poverty by personal effort. An AI future won’t allow it by any means other than collective effort, so that’s what we’ll do.
That will entail an immense amount of violence to be undertaken by the working class against those in power, unless you think they'll gladly give up all of their ill gotten gains out of the goodness of their black little hearts.
In fact I don't think that rich people are cartoon villains who enjoy watching the poors starve. If AI leads to greatly increased productivity, then at existing tax rates there will be more than enough to provide good living conditions for people who can't find jobs.
so why do we have the poverty and inequality today?
why do they spend the money they do on the things they do?
they are not altruists and they never will be.
they could change millions of lives today, but most do not.
pure naivete.
Poverty and inequality keeps declining all the time thanks to economic growth.
In some countries, especially less developed ones, poverty levels are declining, but in the US the trend seems to be the opposite. The middle class is being gutted and descending into survival mode if not poverty, while the top few percent become obscenely rich at their expense.
In any case historical trends are really irrelevant to the discussion, which is about the impact of AI, which threatens to take away almost ALL the jobs (this is what Dario's essay is assuming), which has no historical precedent. Some people like to bring up previous waves of job automation such as the industrial revolution, but the difference there is that automation took some jobs but created others. In the case of AI, AI will also be taking the new jobs that it creates.
If AI takes all the jobs - which is what the people like Dario who are creating it assume will happen - then it seems "UBI" is indeed the logical conclusion (other than those able to make a living working for themself, or via self-sufficiency), but this is not going to be utopia where we are all idle rich practicing our hobbies. What it really means is a welfare state, where formerly proud people capable of supporting themself become dependent on government handouts. What comes to mind is the movie "Soylent Green", not utopia.
I'm not sure how anyone imagines this would actually work. Is there any private enterprise left at all, or are all the means of production (AI datacenters and robotic factories) all controlled by the state. Instead of distributing Soylent Green, the state distributes UBI-scrip, essentially food-stamps, that can be exchanged at state stores for provisions?
Seriously, how could this actually work ?
An alternate future, no more optimistic, at least in the short term, but perhaps more realistic, is that in the fairly near future when unemployment and public pain becomes high enough, we reach a tipping point, and the pitchforks come out. Eventually the government concedes that AI is no more conducive to the public good than nuclear weapons, and heavily regulates it, banning the use of AI to replace jobs. This may sound extreme, but surely not a fraction as extreme at the UBI-based welfare state that Dario Amodei is fantasizing about as the best possible outcome (that is compatible with himself becoming enormously wealthy).
> In some countries, especially less developed ones, poverty levels are declining, but in the US the trend seems to be the opposite.
This *seems* does a lot of heavy lifting here. That's not what data shows.
> The middle class is being gutted and descending into survival mode if not poverty, while the top few percent become obscenely rich at their expense.
The middle class is getting smaller because people are getting richer and start to qualify to higher class, which is opposite of what you just said.
The median income in the US is $84K. Try feeding/housing a family on that, saving for a new car, paying for college education..
The US has the 2nd highest poverty level of all OECD countries, second only to Costa Rica.
> No, I think people will demand to not live in poverty
What about other countries in the world where people... live in poverty?
> Capitalism allows people to escape poverty by personal effort
Isn't it possible that there is a future where AI makes the average value of human labor (or, "personal effort") plummet? Perhaps capitalism will lose some of its edge against a technology like this.
do not seem to have found any possible positive outcome to present
See "Machines of Loving Grace" by Dario Amodei: https://darioamodei.com/essay/machines-of-loving-grace.
This seems more a fantasy than a considered likely outcome. He basically admits that humans will eventually mostly all be out of a job, replaced by AI, but then says (Gemini's summary) that there will be:
"Massive Economic Abundance: Because AI will exponentially grow the total economic pie, overall resource scarcity will diminish. The fundamental challenge shifts from producing wealth to distributing wealth."
So how do we go from everyone out of work, no income to spend on food, or the goods and services that the AI is producing, to "massive economic abundance"?!
It's like the meme:
Step 1: Create AI
Step 2: AI takes all the jobs
Step 3: ???
Step 4: Profit! (massive economic abundance)
What is step 3?
> What is step 3?
"Massive Economic Abundance" implies massive increase in produced goods. This implies massive deflation, ceteris paribus. So step 3 could be simply printing money to pay for UBI. Deflation from AI productivity increase and inflation from UBI money printing will cancel out.
This doesn't make any sense.
For money to work it has to represent some real value, something that has some scarcity to it such as potatoes or hours of human labor. Ultimately it is just a decoupler in a barter system, a universally recognized IOU.
Why would someone give me a car in exchange for UBI-scrip when that UBI-scrip has no inherent scarcity or value and can be produced in infinite supply by the government ?
> Why would someone give me a car in exchange for UBI-scrip when that UBI-scrip has no inherent scarcity or value and can be produced in infinite supply by the government ?
UBI script will have some value, I am proposing printing enough money just to combat AI productivity-induced deflation, not infinite UBI money.
You know, I think Dario should have kept the Richard Brautigan theme in his essay titles. I think that I was trying to describe you to someone is an absolutely perfect match for introducing an AGI that has been achieved. It’s basically the same idea in blank verse, about a spread of a different technology, and made personal along the way.
Asking politely is not how we got a 40-hour work week or workers' comp or most other labor standards we take for granted, just historically speaking. I think a positive outcome is very likely, and I think it will be a lot more work than loudly yelling, but I don't think anything will happen if we try to build everything up from first principles instead of taking a moment to consult history as many are wont to do in this AI era.
What pressure is there to push for any of these changes? I see AI advocates discussing the concept of UBI, but I can't imagine a world where the United States would ever pass this sort of legislation. I mean, congress can barely pass a budget each year.
If you are correct, I expect corporations to reap massive profits while most Americans try to find a way to survive in a world where they are obsolete.
There is whatever pressure you bring to bear on it. As long as you are still living in a democracy your vote is the pressure you can apply. If the parties that exist won’t represent you, make new ones that will. Do something rather than deciding it’s both impossible and up to other people anyway.
I also don’t understand why UBI is desirable. Putting everyone on welfare means everyone is poor. This won’t end well, and it’s certainly not the case that 99% of the population will let a tiny number of people remove their income in favor of pennies for all. Political violence will come first, easily.
No idea why anyone would downvote this, it's true. As someone else pointed out "b = basic." And does anyone think it will adjust with inflation?
Because the implied alternative of the comment is that people will carry on having jobs, which breaks the entire premise that there is AI that replaces all human labour. The comment is missing the point - UBI is the only suggestion anyone has so far that meets this fundamental question about the technology. We will be less poor than without UBI, unless we come up with a better plan.
Isn't it obvious if AI automates your job away, the AI company is going to reap the economic value, which is going to be less than what you cost, while giving you a pittance as UBI? If you received its full value there wouldn't be any point in anybody replacing you with AI.
The only way to win is to wield the AI.
Agreed, a LOT of UBI advocates gloss over the "B" in UBI. If AI increases human productivity overall, the only morally acceptable outcomes (imo) are that everyone's standard of living increases (or at least is the same without having to work) and wealth inequality decreases (if AI is doing ~all the work, there really isn't any sensible justification for some people having significantly more wealth than others). Frankly anything else seems like a recipe for massive social instability.
Nationalised LLM? As long as the state doesn't decide what information the LLM shares (from an output and privacy perspective).
Why should IP persist in a world of “intelligence too cheap to meter”?
I’ll be the first to call myself cynical.
> In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
> We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
UBI in the United States is never going to happen in time. If it happens at all. We don’t even get universal healthcare. I think people who think AI will be a net positive for humanity are also in some sort of denial.
In a different US political climate I would entertain it. If these frontier labs weren’t so clearly going after the money, I would entertain it.
LLMs are clearly a step up for capitalists so I just can’t see any inclusion of LLMs move towards more progressive ideologies.
I guess we'll have to fall back on the plan of hoping one of the 7 Anthropic co-founders has secure power in this world, and decides to use EA principles to give us all a bit of their share of the future light cone ;) (The wink because I'm kinda not really joking that this seems the best plan, if we actually get superintelligence from a US AI company)
Everything you're saying is oddly simultaneously specific and hand-wavy at the same time. I say this because I'm not sure a lot of these things are solvable or, if they are, it's going to take lifetimes. For example:
> How do we replace the work ethic that tells us we are our jobs and idleness is immoral?
For many people it has nothing to do with morality, it's hardwired into their instincts. They want to work, and they will work.
For a lot of us who are not excited about this future it's that no one is trying to answer all the questions you laid out. Instead we have the disgusting people at the helm purposefully spreading doomerism and saying, "We'll figure it out." I think it's pretty problematic (to say the least) to care more about technological advancement than how that advancement is actually shaping up to effect people in the short term. But I know many people don't care, especially those who believe they won't be among the affected.
Beautifully put. This exactly summarizes my feelings about this particular tech. I actually find it hugely useful. But I fear we lack the wisdom to use this in a way that won't destroy us.
I don't understand why this got downvotes - simply extrapolating current trends leads to the need to answer all of these questions.
My own $0.02 on the economics piece - every country should have a sovereign wealth fund. Governments should block market access from automated[0] companies until those companies provide equity contributions to the wealth fund for that country. This aligns regulator and corporate interests. Dividends flow into the sovereign wealth funds and then can be allocated locally from there - UBI, job programs, etc. Let different jurisdictions explore different ways to structure a post-labor society.
On the broader social front - I think a lot of lack of meaning discussion boils down to the overemphasis we have on your job as your self-worth. We need to realign our societal expectations - and people need to spend more time with their families.
[0] for this to work, I think we would need well accepted metrics for 'how automated' a company is - and that probably needs a 3rd party auditing industry.
Nationalize the data centers, reserve enough inference to automate power generation, food production, transportation, and housing.
Is there any government that has gotten socialism correct for its citizens? I'd point to UAE/Qatar if they didn't depend on human servitude and inequality.
I would not advocate nationalization of any operations - I think the free market will run the data centers better in general.
I'm only suggesting (eventual) negotiations between countries and automated corporations - to operate here, you contribute equity.
Why do you think the AI companies are going to let it be a “we” kind of decision, and since it’s obviously not going to be a “we” kind of decision, as getting to this point certainly has not been thanks to people like you, what makes you think living standards are going to be broadly uplifted?
[dead]
[flagged]
The human zoo where the top ~250,000k humans live in a "human utopia" and the AI provides while mostly focusing on whatever it decides it's own goals are.
Humanity survives (but we reading this probably don't), the AI treats the living humans like the Emperor's favorite pets (probably a pretty good life), and then the AI does whatever else it deems important.
To what kind of goal that an ASI might decide to pursue would "a quarter billion happy, healthy, free people" be the most efficient solution to?
My mistake, I meant 250k*
[dead]
Realistically it means trying to start a small business of some sort, because AI is hugely advantageous to business owners and disadvantageous to workers. And it's something AIs can't do unless they get legal personhood, which may well not happen any time soon.
Hopefully doesn’t ever happen, since that’s one of the more plausible omnicide scenarios. I human like AI is almost certainly achievable without much research effort at this point, but we shouldn’t do it.
[dead]
[flagged]
> "probably only 20% chance we all die"
Bad news for you -- there's a 100% chance we all die. Sorry to be the one to tell you.
Logical mistake. There is a difference between every human dying eventually and humanity going extinct. The former doesn't imply the latter. The previous commenter clearly meant the latter.
What makes you think separate nations would effectively cooperate in this way?
Might as well try, no? Or do you think it's better we just risk it? I mean it's not like the West has any hard or soft power to encourage cooperation.
Your 'serious' proposal is unlimited global military bombing campaign on civilian infrastructure by the United States (which is currently losing a war using the same strategy) to 'solve safety' preemptively against a 20% number you just made up?
And you accuse the 'other side' of 'suicidal apathy'??
You should put down the AI and do some self-reflection on how you came to hold these views.
> You should put down the AI and do some self-reflection on how you came to hold these views.
I've been in the field for almost 2 decades, and actively thinking about AI risk for over 15 years.
I've always held the controversial opinion that there may come a time where we might unfortunately have to consider using force to protect our civilisation from the threat of ASI, but I've always reframed from openly discussing that because I'm so personally against any use of force and there's always been time for more peaceful options.
Several months ago I changed my tone on AI risk to stop worrying about the optics of what I'm saying because it's too late in the day for me not to speak plainly at this point. Similarly over the last week I've decided I can no longer reframe from advocating for the use of force (if necessary) to stop rouge actors from playing Russian roulette with civilisation.
Please understand I don't want anyone to be harmed. Perhaps you don't agree with my suggestion, but my stance ultimately comes from a position of harm reduction.
> preemptively against a 20% number you just made up?
It's a probability estimate. Happy to expand in detail on why 20% specifically, but first you accept I can't tell you what's going to happen in the future with certainty right? To some degree any prediction you might have about the future will be "made up".
Advocating for nuclear war as a means of 'harm reduction' should be causing you serious cognitive dissonance.
There are dozens of escalating, yet fundamentally less destructive means to prevent a data centers from continuing to operate (if that's the goal). Not starting at those more modest interventions is political malpractice and in opposition to your harm reduction narrative.
Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.
Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.
Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.
> Advocating for nuclear war as a means of 'harm reduction' should be causing you serious cognitive dissonance.
If you're referring to the "nuke the data centers" line in my bio, that's tongue in cheek. It's an expression of my sentiment towards AI at the extreme.
To be clear, I don't think any country should seriously be considering dropping nukes on data centers. Targeted strikes on data centers operated by rogue private companies or nation states is acceptable to me – assuming pre-warning of intent and appropriate regulatory foreshadowing.
I suspect we agree this is reasonable?
> Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.
> Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.
No, they're not based on unbounded exponential growth curves at all... I'm actually somewhat skeptical of claims that there could exist an intelligence that far exceeds our own. My assumption is really just that AI will be far faster, far more knowledgable, far more persistent and marginally more generally intelligent than the average human. The first three are already true, the forth we're getting to.
We're on an exponential curve right now and I suspect the rate of progress will accelerate some more as a result of RSI. This will likely continue for 2-3 years (if we have that long) before starting to flatten out.
As for my doom prediction, my median concerns are around around the destabilising nature of ASI.
If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation. I can go into more detail here if you wish.
However, I'd still put these destabilisation risks at under 50% in total. My concern more broadly is the majority of ASI outcomes are doom scenarios. It's extremely hard to imagine a world in which you have a magic box that can grant any wish that intelligence can grant, but which we have a functional civilisation. You have to basically assume all potentially dangerous wishes won't be granted and probability the only feasible way to do that is restrict who can make the wishes – but then you have the concentrate of power risk.
I largely agree with the premise "If Anyone Builds It, Everyone Dies", so we must try to avoid building it. I think we can build ASI safely, but we're clearly not on the path right now and I suspect it might take decades (or more) to figure out how we can build ASI with an extremely high probability of a good outcome.
> Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.
I do this. I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.
I don't think we realistically have a chance of diverting from this path at this point. I have made peace with this but it makes me very sad and haunts my dreams almost every night.
> If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation.
We already live in a reality with unprecedented concentration of wealth and power in the hands of the very few. This process has produced horrific violence and ended civilization for tens of millions.
The actual impact of AI, so far, has been redistributive upwards (solely through speculation, not via productivity or displacement).
This kind of problem is a standard human social ill. Its solution is social too.
Redistributive economic and political action can equalize power and wealth disparity. AI is incidental in this story.
Your understanding of AI as an isolated perfected intellectual djinn lacks a consideration of the existing and ever-changing labour and social relations that give AI responses meaning in the first place.
It is a compressed, multi dimensional manifold containing the structure of relations between objects. Responses are a reification of a point in that space.
Navigating the space can be interesting, but it is a poor substitute for the relations themselves.
Very few people yearn for the endless, frictionless solipsism silicon valley promises (AI + slave robots), and those that do are overrepresented in your social circles. The books they write are a study in psychological projection and limited political education.
> I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.
This is good advice generally. But it isn't organizing. Join something, meet up, drink bad coffee, listen, take notes, advocate to fix just one bad thing in your community, feel your impact reverberate through the world, and then notice how much easier it is to shake off despair.
If you don't have a community, then no wonder everything feels like its ending.
If the thesis that "If anybody builds it, everyone dies" is true, or has any chance of becoming true, then it is the logical thing to do. As Geoffrey Hinton said, "If you want to know what it's like not to be the apex intelligence, ask a chicken."
It is not 'logical' to imagine a hypothetical doomsday scenario that justifies a preemptive nuclear war. (which is what the grandparent commenter's bio contemplates, 'nuke the datacenters' is their central credo).
Does it bother you that the people who are publicly cocksure that P(doom) is moments away are the same people that have profited most handsomely from that pronouncement?
That the 'humanists' that want to do 'altruism' for 'potential future humans' and are the same people that commit fraud and theft at a civilizational scale, then sell this 'intelligence' to any child-incinerating militaries with spare cash?
It's not wrong to want to do good, but if a system that is branded 'do (the most) good' commits great evils, you are morally and intellectually obligated to step back and reconsider how you are spending your time.
Also, I asked a chicken and a feral rock dove what it's like to be not be 'apex' and they burbled at me and kept eating millet and sunflower seeds.
Would you like me to follow up with them? I'm not sure what point you expected them to make.
> It is not 'logical' to imagine a hypothetical doomsday scenario that justifies a preemptive nuclear war
You hallucinated the "preemptive nuclear war". He didn't say anything about nukes. That's your own invention.
> Does it bother you that the people who are publicly cocksure that P(doom) is moments away
20% is not "cocksure". The "moments" is again an exaggeration.
I did not, I read kypro's (the OP I was replying to) bio, to whit:
Every problem is a search problem. Nuke the data centers.
P(doom) = 98.9% (Aug-2026) P(doom) = 98.2% (July-2026) P(doom) = 98.2% (Jun-2026) P(doom) = 98.5% (May-2026) P(doom) = 98.8% (mid-April-2026) P(doom) = 98.7% (April-2026) P(doom) = 98.7% (March-2026) P(doom) = 98.5% (mid-Feb-2026) P(doom) = 97% (Feb-2026) P(doom) = 94% (Jan-2026) P(doom) = 93% (Dec-2025) P(doom) = 95% (July-2025)
I've removed the "nuke the data centers" line now.
It's was an expression of my sentiment, not a policy position I'd support. I wrongly assumed that was obvious, but some people are making bad-faith assumptions about me and my sanity.
Not that it should need to be said on a comment thread where I am express concern about civilisation in a post-ASI world, but I obviously don't want to see a nuclear war.
If the Judeo-Bolshevist theses were true, then Operation Barbarossa was the logical thing to do, and everything which came with it.
Powerful word, `if`. "You're not only wrong you're a fulminating psychopath" is a perfectly valid response to getting it wrong like a fulminating psychopath.
GP said cooperation, never implied the US would do it alone.
No.
You’re not necessarily wrong, but math proofs aren’t going to unify today’s brainrotted population.
> We are now at the point where RSI is feasible
- Hitchen's RazorThe evidence is abundant and nearly impossible to ignore without increasingly focused effort.
So abundant that you won't link anything? Send a peer-reviewed paper then, let's settle this.
Did you read the article?
AI is a math & computer science problem. If AI can do advanced maths and computer science research, then it can begin to suggest useful algorithmic optimisations.
Right now I'm sure the vast majority of these will be junk, but occasionally, even with current limitations, they might occasionally stumble on something.
It's not really whether RSI is or isn't possible, it's really just whether it's the most efficient way for labs to improve their models today given they have limited compute to run AI-generated experiments and access to very intelligent humans who might have a better hit/miss ratio.
Do you disagree with anything I'm saying here? Do you not think current AIs can suggest algorithmic improvements or something?
There are some genuinely insane takes in this thread on both sides but this takes the cake.
A few years ago the majority of Hacker News dismissed LLMs as not much more than stochastic parrots who were decades away from doing any serious work, autonomously escaping containment and hacking Hugging Face, or outperforming most professional mathematicians at proving theorems. These things were labeled "insane" and "science fiction" despite the rapid progress we had already seen.
Now you are again postulating that there would be no more extreme progress in the near future. That's actually more "insane".
Being right about the thing that’s genocides all culture and likely worse isn’t such a great thing to brag about, but go ahead.
That's because its satire, the tell was "not excluding targeted military strikes"
Gonna say the edit they made removes the satire possibility.
Not satire, but as you can see one is engaging in good faith and everyone is either engaging in character attacks or low-effort counters.
It's fine, and I'm very used to this, but if you want to have a dialogue I'd be very happy to. I'm a very reasonable and sane person outside of apparently holding some controversial views on AI risk =)
I'd love to know why you think what I said was insane?
Which straw man are you arguing against?
I don't think this is a straw man, a huge number of people in my life (non CS people) think AI is a dead-end, that it's just a stochastic parrot, that it'll never be able to do many things that humans can do. I have had many arguments with people who told me that "AI will never be able to do X", and then 6 months later AI is able to do X. Then they will move the goal posts and say "well AI will definitely never be able to do Y".
[dead]
[flagged]
I can deal with apathy, that’s the norm. What bothers me are all the people who think they can suppress AI by talking it down. That’s what’s counterproductive, just pretend the problem doesn’t exist. Tell other people it doesn’t exist either. I get it, it’s threatening socially, economically, maybe existentially. It’s also not going away.
So you think it’s a potentially existential threat but are bothered by people who maybe want to suppress it… Hopefully you acknowledge there is a bit of lack of self awareness here eh?
You seem to have misunderstood my point.
It’s not apathy. It’s the fact that almost nobody can really understand what these results mean.
I’m not a mathematician so I have zero clue what “ New upper bounds on sphere-packing density down to the Cohn–Elkies thresholds” means.
It is a fact of experience, and indeed effectively a theorem, that the better they get at coding and math, the dumber they are. These are the wages of RLVR etc
completely false
Sorry to hear you've been impacted by this AI math.
I heard that Gary Kasparov was impacted by AI chess, but at least he still seems to have a job, so don't give up.
Making insulting assumptions about the hidden motivations of others is not the level of discourse I come to HN for.
FWIW I was responding to the sarcastically expressed overt message that AI-math is a clear sign that AGI is here and anyone who disagrees is in denial.
My point being that AI math is a narrow skill just like AI chess and implies nothing about generality (AGI).
Sarcasm begats sarcasm.
"AI" has limits in that it cannot invent knowledge, it can only distill and search for patterns in existing knowledge
not sure how many will get this reference but "AI" for science and math is like super-shoes for runners
at first we are blown away by the impossible improvements including sub-2-hour realworld marathon and every other PR/CR/WR is dialed down
but then the improvements slow and reach a stall point because of the limit of technology and the source of the achievement
ie. sub-2-hour marathon yes, sub-1-hour never happening (rollerblade inline-skate record is 1-hour marathon)
I agree. The most likely scenario is that this is just a "new normal" lift that is percolating through human endeavors and will saturate at some point. For example, the whole cyber-security bruhaha should ultimately resolve into higher standards for code published -- we can now cheaply find and fix a whole slew of minor bugs that weren't worth our time before.
The fact we see a lift is not the same as evidence that the lift is unbounded.
The lift being finite is supported by the fact improvements have come at the edges: improvements from human feedback, improvements in harnesses, improvements on model compatibility with harnesses, improvements in inference efficiency with new architectures, etc. If we were just training better models from scratch that would be one thing, but we are just making better use of a tool we've developed.
Not sure what your first sentence means, or why you are quoting AI. Many of these problems individually were math at a level approaching the highest possible for expert human mathematicians. These are not simple combinations of existing ideas, or following of human intuitions, or implementing something following specific human instructions. Then again maybe you mean that math is not knowledge and that all math is simply extending the basic axioms using known patterns, to which I would not agree.
as much as I love analogy with humans using super-shoes, not everybody can be a world-class expert in their industry. There should be a place at the table for average people to take part; otherwise, it won't be sustainable.
As a programmer, I am mostly interested in whether my role is sustainable long-term and whether the models will get better. I don't feel in jeopardy yet, but two more years like this and the calculus of hiring software engineers could shift even further. QAs are already overwhelmed with work
well not every runner can run world-class sub2 marathon or even vaguely close, not even in super-shoes
but with super-shoes more and more runners are qualifying for boston marathon and even olympic trials marathon where it would have been impossible for them previously
and that's what "AI" currently does, it allows average people to immediately "pick the brain" of every expert in every field, in every scientific paper, without previously reading a single other google result, something that would have been impossible for them previously (super-shoes for the brain? too far?)
but "AI" isn't creating new knowledge, it's just stitching together existing knowledge from patterns that would have taken years by human hand if even possible at all, it's going to "hit the wall" eventually (in its current form)
How do humans "invent knowledge"? Is your argument that the answer to these questions already existed in the training set? Why didn't any human recognize that before?
humans discover a lot of things by trial and error (aka what I meant by "invent knowledge")
basically everything Benjamin Franklin did was trial and error because no-one understood what electricity was in the slightest
almost everything Edison did was trial and error too, he had his lab try thousands of materials for his long lasting lightbulb filament
even the most advanced "AI" today is just machine-learning going through everything already known trying to piece together previously discovered facts, admittedly at levels and detail impossible by human hands
but that means there are limits and it's not really "AI"
Have you seen how it debugs? It's trial and error. As the most reasonable hypothesis fail, new data is found, and it keeps iterating. It's not a human intelligence, but if it has an issue, it's not some mysterious creativity anima in the heart of man.
I have seen it produce tentative genetics for experiments, just like a scientist does. Then the data comes in and it can evaluate the data from the experiment just as well. One just had to give it a lab budget.
Any computable problem will eventually fall to computers.
LLMs have made math proofs more computable, in the sense that a computer can both generate potential solutions and check the validity of its solutions on its own, with a reasonable chance of converging on something correct. I assume this was already doable to some extent, but it seems like it’s now exponentially easier. That still doesn’t mean that all math is automatically solved.
This is somewhat similar to things like molecular dynamics or protein folding or finite element simulations, etc. Some problems that were previously intractable via computation became tractable. Others - the vast majority of other problems - remain unsolvable by these computational techniques, because the scale of compute required is beyond imagination. These are simple things like simulating the dynamics of a cubic millimeter of water molecules for 1 second. Unfathomably beyond current capabilities (and LLMs aren’t going to change that).
I think LLMs are great, I use them every day and I think they have a ton of value. But if these things were as revolutionary as people promote/fear them to be, you should immediately point them at the highest value math problems and see progress. Like the Millenium Prize problems. Haven’t seen a solution to those.
So there are limits - but we’re about to learn a lot about the new normal of what constitutes a layup math proof vs the truly difficult.
> Any computable problem will eventually fall to computers.
I think the question, that we keep stumbling over, is what problems are computable.
> But if these things were as revolutionary as people promote/fear them to be, you should immediately point them at the highest value math problems and see progress. Like the Millenium Prize problems. Haven’t seen a solution to those.
Let the goalpost shifting continue. It'll buy us another half year or so.
If by solution you mean a proof and by testing you mean encoding it in lean and compiling it, the space of possible syntactically correct proofs which you can encode probably explodes in a way that is well beyond what any computer could try to brute-force. LLMs don't brute-force proofs, i believe their approach is quite similar to humans. I believe the same is essentially true for counter-examples of the type that have been found latelly, they are not found by search, but by using theory.
On the other hand even if the compute allocated by openai is esquivalent to day 10 human mathematicians, the machines can work 24h per day, that is already a lot more productive.
In my opinion, after seein the Jacobian conjecture go, the Riemann hypothesis only has about a year left.
I’m not sure if people just aren’t as aware, but the Jacobian conjecture practically was on par with those other great problems.
Whether or not it’s on par famously or difficulty level doesn’t predict whether the others will fall. They’re unique problems and math isn’t linear - the Jacobian it managed to find a counterexample and relied on other proofs that had been developed showing >3 case == 3 dimensional case. However, the Riemann may not fall in the same way because it may actually be true or the surrounding math isn’t quite ready to tackle that problem.
I would be genuinely very impressed - but still not scared - if the Riemann hypothesis were solved. I suspect that we may require “new math” to make progress on that. If a new operator / symbol is required, is that fundamentally not doable by an LLM because it’s outside of current tokenization space?
Obviously not, there's no need to create a new character for a new operation. You can just define it as @ or or any other symbol you want.
"Any computable problem will eventually fall to computers."
By definition. Its those pesky NP jobbies that get in the way.
Science is not merely computing though, even the theoretical sciences
Just to re-iterate the point:
Whenever the handwaving starts around a discussion relating to a NP hard problem, I find it useful to imagine a Canadian bloke (MHRIP) in a red top, with a ... Scottish accent ... saying:
"Ye cannae break the laws o' physics, Jim". (maffs not fisics, obvs!)
If that is a bit tiresome for the gung-ho AI evangelist, there is also the rather knotty snag that that blasted Austrian geezer Gödel fiddled up: incompleteness.
Its almost as though these bloody clever scientific and that types keep on putting artificial blocks in the way of LLMs laying golden eggs!
I'm quite happy with the "marginal gains" I get with a DGX Spark. It will pay for itself within three months doing stuff on prem and us not sending data to someone else. It will scale.
They don't need to make NP-hard tractable to shake the world order.
They just need to be better than humans.
Would be great to see them solve Yang-Mills and Mass Gap.
Not being an expert in any of the fields OpenAI has "advanced" I don't want to prematurely downplay the significance of this contribution. However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing.
It is true there hasn't been a reliable computational approach to solving these problems before. But do these proofs contribute new ideas to the mathematical corpus, or are they simply an effective method to exhaustively search the literature for the right combination of existing tools to apply to the problem?
Essentially, did these problems seem like they had an intuitive answer and were feasible to prove before, just not high enough value targets for an expert to invest time into? Or were they fundamentally difficult prior to this point and it appears that AI has done something more than just throw the problem into a big solver.
The ones I'm familiar with are big breakthroughs, but they are both counterexamples. Examples have an advantage in that once you have the example in hand and a sketch of the proof (which they have provided), then an expert can probably work out the details themselves.
The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more.
The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.
> but proving a group was non-sofic was out of reach
a colleague was telling me that the base idea for proving that something is not sofic already appeared in the literature around 2019 or so (this is the "expanders graphs" that are mentioned in OpenAI s paper. no one had managed to find a concrete example though. this doesn't make the result less impressive in any case.
Interesting. Do you have any more specific insights into where you feel AI was a big value-add to these problems? I don't want to be overly dismissive of AI, but I also feel that the AI hype engine frequently positions claims as being 'ground-breaking' when they are really just interesting incremental results.
The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.
They are more than a strong student could achieve. I'm not equally familiar with the problems, but the ones I'm familiar with, if a student solved them people would be thinking "that's someone on track to win the Fields Medal one day".
If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."
The problems from CS (CVP and circuit complexity) are very important problems that have been worked on by top researchers for 30-40 years. Some of these researchers include Turing Award winners. A solution to them would be a best-paper award at many top CS conferences.
I assume you're talking about No. 5, the arithmetic circuit complexity bound? The existence of a lower bound than state-of-the-art is certainly a significant result and worth publishing.
But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).
Circuit complexity lower bounds (and lower bounds in general) are notoriously difficult to come across.
For example, despite our best efforts, the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n). In contrast, our best algorithms for the task run in time roughly O(2^n). That’s an exponential gap. This is despite decades of trying to find lower bounds.
> the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n)
Wow, that’s pretty stark.
“What’s the minimum time it would take to solve this problem?”
“Well, at the very least you’d have to read the input the whole way through”
> However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing
Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.
I have no idea how many PhD's have spent how much time of their careers tackling these very specific problems, and I doubt you do either.
I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.
They are significant problems which experts have spent months or years studying. I heard a mathematician say that resolving non-sofic groups and Connes's rigidity would be career-defining for a mathematician.
"Breakthrough research" can be defined (in the citation record) as research that both (1) becomes highly cited, and (2) brings together citation chains that were previously not showing up together.
Mundane incremental research is cobbled from existing citations that already appear nearby in the record.
Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record.
So we can know pretty conclusively.
Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on)
[1]: https://www.youtube.com/watch?v=guLDNMAOn24
Breakthrough math research is very rarely highly cited. Maybe some combination of pretraining scale, inference speed and orchestration will help, but it's telling that OpenAI is solving random math research problems rather than bedrock algorithms and their implementation. Even as cool as the tech is, there still is very much a clock that they have to outrace before they collapse.
I'm not arguing that this isn't innovative or worthy of publication. Basically any result that moves the needle meets those criteria. I'm interested in how the results that OpenAI has published here differs from finding optimality solutions for incredibly niche optimization problems by throwing the problem in an enormous solver.
They differ in that there hasn't been a solver you could have thrown them at. I guess you could argue their harness + LLM setup is a "solver", but the approach is so different from what we used that word for in the past that I don't think it would be appropriate.
You’ve received the expert answer several times. You just don’t seem to like the answer.
Forgive me for taking everything salesmen say with a grain of salt.
It’s a marketing. They are a sham company. If this article was by Scientific American or something it would be worth a lot more. They are literally trying to keep the hype train on track.
Also on HN front page today: AI's debt binge can't last, hidden borrowing reaches $1.65T (fortune.com)
https://news.ycombinator.com/item?id=49160699
It is marketing, they are a shady company, and yet, if someone had access to these results before today's modern AI tools, they could get tenure at any university in the world.
As others have said, it's hard to know how significant these results are without more transparency around the methods used to obtain them.
Not really. Why would that be the case?
If the prompt was crafted by an expert mathematician with pages of context and insights provided to the LLM to put it on the right path, that is a lot less impressive than a prompt that just says "find a counterexample to this conjecture".
Similarly, if they are spending millions of dollars in inference just churning on thousands of problems and these are the 10 solutions they came up with, that would be less impressive than if they chose these 10 problems specifically and were able to come up with solutions.
one of the early premises of how ai takeoff would go was that a system that could solve open problems in advanced mathematics would also discover novel advances in math and computer science that directly unlock drastically better software performance. we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B). we are also seeing incredible advances in software performance. open ai announced like 15% improvement by fixing gpu kernel issues. these are clearly linked in the sense of scaling laws and generalization of intelligence: a huge model gets capabilities in both math and software engineering that isn't possible at smaller scales.
but it seems less likely to me than before that the types of math/science discoveries will explicitly unlock better software performance. in some sense this fits our intuitions. when top tech companies use math PhD type employees, they have them stop doing pure math research and instead focus on software engineering. these people are often very good at software engineering but not due to recent discoveries in academic mathematics, it's due to their general intelligence. to me, this is evidence that the models are getting better but does not make me think we are on the cusp of a foom style fast takeoff enabled by revolutions in frontier math (i also posted this on twitter @mlipman13)
I don’t really like AI but let’s stop kidding ourselves, no human mathematician could make progress on a dozen major open problems in a week or two. If you’re measuring it against humans then it is by far the best mathematician to ever live.
I guess it depends on how to measure a single "person"? If you spun up 2000 copies of Terrance Tao, I wouldn't be surprised if you found a few new discoveries at the end of it.
The impressive/surprising thing is your premise because we effectively can spin up an army of mathematicians now
> I don’t really like AI but let’s stop kidding ourselves
If I had to create a tagline to describe my opinions about AI in a single sentence, that’d be it.
It’s possible to both hate AI and be impressed by it at the same time. Lying to ourselves about its capabilities does us no good. It’s emotionally difficult to do, but people need to come to grips with what’s happening and shake themselves out of a state of denial.
How much investment has gone into OpenAI versus mathematics research in 2025 for example? Probably 100x?
The AI results are clearly impressive. But these sorts of things are also in the ballpark of what human effort could solve given enough attention and time. Though it is hard to say.
This makes no sense. To believe this you have to think that the models are somehow being overfit explicitly on academic mathematics and it doesn’t carry over at all to more practical software engineering. I wouldn’t make that bet.
Unlike math, software is constrained by the physical world.
In what sense?
For example, pi can be computed to arbitrarily many digits, but could only even in principle be accurately represented with physical objects to a precision that many humans could memorize easily. This is thanks to physical constraints such as "diameter of the observable universe" and "Planck length", at a minimum.
Even if pi is an approximation, it is still representable mathematically. Floating-point arithmetic is an algebra (in this case, a magma [1]), which can be studied mathematically.
[1] https://en.wikipedia.org/wiki/Magma_(algebra)
This also makes the assumption that frontier math has all the long hanging fruits already taken... also very dubious.
Some of the problems solved here, at least in CS, have been open for decades, and have been worked on by very smart leading researchers in the field, including Turing Award winners.
Like, these would be best-paper awards at many top CS conferences.
Or, that the mathematical formalisms that model the limits of software performance are firm enough that barring P==NP, nothing much will change despite proofs of beautiful math.
> we are also seeing incredible advances in software performance
Incredible?
> open ai announced like 15% improvement by fixing gpu kernel issue
That is... ordinary software optimization.
a 15% improvement at a trillion dollar scale company is massive
15% improvements are usually called “fixing a mistake in the code” or “getting to that task in the backlog for optimizing that code we had to ship on a deadline”. They’re more likely the bigger the company: more contributors working in disparate areas means more low hanging fruit is probably lying around.
There's nothing ordinary about downloading a new GPU driver and having performance go up by 15%.
This is untrue. It is very ordinary. How much do you know about GPU drivers that you state this so assuredly? Drivers are software. Software can be improved. Do you believe there are no prior examples of GPU drivers being improved such that particular compute patterns go up in performance by more than 15%? This driver improved Total War performance by 71% https://www.nvidia.com/download/driverResults.aspx/74714/en-... Also note you can go ahead and improve any open source driver right now, most likely. Compile it for your specific card and remove all other architecture specific if-cases and you can get an improvement.
Edit: also here’s a opencl 30% compute perf increase documented here : https://m.hexus.net/tech/news/graphics/74425-haswell-systems... that i just googled for
No, we do this kind of thing regularly. Optimizations often depend on the model that is using them. Namely, fusion techniques.
Correct me if I'm wrong, but all of the aforementioned advances were made in the last year? Until very recently few people had access to these tools. Most people still don't know how to use ChatGPT, and very few use tools like CC regularily. If in a few years these frontier tools become commonplace and people upskill we would should see a network effect?
> but it seems less likely to me than before that the types of math/science discoveries will explicitly unlock better software performance.
You seem to overlook a simpler barrier. To make these advances, they have to be possible. A 15% improvement in GPU kernels doesn't evidence that significantly more improvement has been left on the table.
Google has also invested a lot of time into developing new hardware and new algorithms with AI (other types of AI, not LLMs). I don't know if it's paying off (haven't followed it closely) but they seem to think it's worth the effort.
This will depend on the problem; I expect big algorithmic performance improvements in AI since the algorithms are still new, inefficent, and constantly being improved. But maybe not for sorting, fast fourier transforms, or other well-studied basic algorithms?
A lot of algorithmic improvement in AI is ultimately bottlenecked by compute. It is very easy to come up with ideas that could improve models! But to prove that they do, especially at scale, is expensive and takes a long time.
> novel advances in math
> we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B)
i think you have misunderstanding of what mathematicians do
> i think you have misunderstanding of what mathematicians do
They get to make cool 3D plot visualizations of functions so obscure to me that they’re named after someone who is still alive - and/or get to work on cryptography for the NSA - I think?
The GitHub repo with the Lean formalizations just came out a couple of hours ago: https://github.com/openai/ten-proofs
It also links to a paper written by an LLM where the model "reconstructs how the proof came together" based on the unpublished reasoning traces: https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf
I wish they'd publish the prompts though!
It seems that a lot of folks misunderstand the guarantees that lean provides.
I just want to state that having "lean proofs" that build (checks) does not mean the actual real theorems we care about hold. Ignoring lean kernel bugs, ultimately a human (not an agent) has to verify the lean encoded theorem statements (specs/specifications) that the lean proofs are checked against. For non-trivial theorems such as these, this is an arduous and tricky task where even a little mistake could be fatal. AI generated lean encoded theorems can be huge and difficult to understand. I wonder if anyone reputable has audited these specifications.
This is the Lean proof that a nonsofic group exists (34,440 lines): https://github.com/openai/ten-proofs/blob/main/NonSoficGroup...
This is an extraction from that of the actual theorem statement (39 lines): https://github.com/openai/ten-proofs/blob/94bc0feb6a9ff12c7d...
It may be for this theorem there's a succinct description, but still needs to be checked carefully. However, there are others that are non-trivial.
In some cases you're right, but I think that's often a symptom of mathematics in Lean being relatively immature (i.e., it will get much easier with time). Even then, verifying the statement in Lean is correct is still much easier than verifying the natural language proof is correct.
Exact prompts haven't mattered for about a year now.
Care to elaborate? Curious about this. Is this because LLMs have been geared towards understanding user user intent behind a prompt rather than following the instructions exactly?
There's a full fledged 'reasoning' step that basically expands your prompt.
As long as you are not missing important information, how you word the prompt does not have any effect.
Isn’t that a huge simplification? Of course the way you phrase the prompt can carry semantic meaning, maybe subtly, but still. And sometimes that matters a little and sometimes a lot. I’ve stopped numerous agent sessions over the last few weeks to reword my initial prompt to get the agent off an unintended track.
> reword my initial prompt to get the agent off an unintended track.
The signal here is the action of stopping the agent “do something else, this is stupid”, not a tweak to the initial prompt.
Oh yeah, I suspected it was something like this. Thanks!
I like the Lean formalizations — I hadn't thought seriously of asking for that before but might try it with some stuff I've been working on.
[dead]
Henry Yuen's (whose work problem 6 builds on) comments on this are worth reading IMO: https://bsky.app/profile/henryyuen.bsky.social/post/3ms2jpch...
This starts to feel like chess engines. It’s obvious their play is superior but it’s impossible for humans to understand the moves.
It's going to be an interesting time if this generalizes to other fields.
It sounds like he hasn't verified the results of a problem that he has personally worked on, so how many of these problems have actually been verified?
From what I understand all of them have Lean proofs/certificates thus are basically 100% proven without a doubt.
Besides for what others have mentioned, the lean proof could be proving something else. Given AI’s propensity to hallucinate, seems like someone should check the lean proof actually expresses what it’s claimed to.
Incorrect. The statement in Lean can itself be wrong. Moreover, they could be exploiting a kernel bug in Lean, of which we had one published literally a week ago.
We recently saw that lean itself isnt proven correct. Its not likely but i wouldnt call it verified if its only verified in lean
https://x.com/gro_tsen/status/2082483878480977959
Between a lean proof, and a peer reviewed paper, the former is a lot less likely to be mistaken...
Nothing is perfect.
I mean, they're verified in the sense that the lean proof checks out... and presumably OpenAI read them.
Or they made another LLM 'read' them?
> You are an expert in the field of mathematics, with decades of experience. You are a reviewer of proofs, etc etc.etc.
Can’t wait for this stuff to have quality of life increases for the average person. So far all I see is that AI has made owning a computer more expensive, made some jobs redundant, increased spam and distrust with questionable authenticity of content and of course made some Americans very rich.
Not everyone works for Evil Corp. I work in the public sector and my work supports public health and safety initiatives. AI has allowed my team to get much more done than we would have otherwise which improves the quality of life of the people in my community.
So I would like to counter your cynicism with a “YMMV” depending on who you work for.
I work in the public sector and my work supports public health and safety initiatives.
What does "support" mean in this context?
Examples?
HN doesn’t like when you ask for those.
Remember, this is the propaganda arm of a VC. Proof is a hindrance to the hype that drives value.
> AI has allowed my team to get much more done
We often hear this. Likewise to get things done faster. And to get things done cheaper.
Almost never to get things done beteer.
Are you just automating lab reporting and results faster? That doesn’t exactly improve the health of others, a faster lab does not cure an ailment or provide a better cure.
Like cool my lung xray only took minutes to determine if I have a lesion instead of a week or a few days, but I still have cancer.
For some people, getting lung cancer reported a week earlier will save their lives.
More importantly, if you can screen for cancer in a way that takes minutes instead of a week, imagine how accessible this technology will become.
Faster scans means more scans. More scans means getting scans earlier and tracking abnormalities on scans over time. This can lead to earlier intervention, which means you get treatment for the cancer when it’s stage 2 instead of stage 3 or 4, which maybe is the difference between living a full life and dying young.
It has had significant quality of life increases for me. I use LLMs for everything from:
- travel and restaurant recommendations. my last few outings have been entirely LLM-advised and they turned out excellent. LLMs seem to have ingested every single Google review, photo, and menu of every business on Earth and can answer very nuanced questions like "is the garlic chicken at <restaurant, city> garnished with coriander?"
- fitness, nutrition, accounting, therapy, medical, legal, immigration advice (sure it's not a real professional but you know what, it's pretty fucking close, and any capability gap is made up by having perfect two-way communication which you don't get when talking with a human)
- coding (work, side projects, personal tools, documentation & pricing questions, "review this code", etc).
- I start reading most articles with the prompt "Summarize this article: <url>". I just started a non-fiction book by pasting into Claude: "There are 12 chapters in the book <book-name>. Can you give me a 2 sentence synopsis of each chapter?". It reduces the "activation energy" hump and screens if it's worth reading at all.
- I use the LLM in my Tesla for on-the-fly advice for parking and other things. You can simply ask "what's the best Boba place around here?" and it will give you a decent recommendation. You can also follow up with "does this place have ample parking?".
- I use the LLM in YouTube to summarize videos and ask specific questions and/or get timestamps to the parts I care about.
If your critical thinking skills are strong then LLM is a literal superpower.
I'm just barely old enough to have experienced a couple waves of technology being introduced and creating these sorts of life optimizations. It almost never actually improves quality of life. It just makes life more optimized, and since everyone else is doing the exact same optimization as you, that optimization goes from novel and fun to table stakes to a basic necessity to compete and function in society professionally, at which point it's really nothing but pure efficiency, not genuine improvement in relative happiness.
1. Do frontier lab LLMs make your rent cheaper? Groceries? Healthcare?
2. Do LLMs reduce the loneliness epidemic?
3. Do they reduce population aging in almost all counties around the world?
4. Do they reduce political polarization?
5. Do they bolster democracies?
6. Do they decelerate climate change and general environmental destruction?
7. Do they accelerate sustainability and the circular economy (not circular financing!)?
8. Do they reduce the workweek and give people more free time for family and hobbies?
Etc, etc.
I would hold off on calling anything a "superpower" unless it solves the hard problems in life. Heck, computers and even the internet barely score better than LLMs when measured against the important things in life.
Actually, you are right. We are working with different definitions of quality of life.
The QoL improvement from LLM is somewhere in the ballpark of going from dumb phones to smart phones for an average person (able-bodied, local to the area etc). It's nowhere near Haber-process or penicillin.
> do they make your rent cheaper
Well maybe not the LLMs, but AI and robotics have a lot of potential here.
https://youtu.be/bsNTv8t239Y
It's still the early days though. Try again in 15 years!
> 8. Do they reduce the workweek
Sure. Often to zero.
I'm talking statistically, not what happens for a lucky few.
Keynes was hoping that our workweek would be 20h/week by now, through technological advancement. Instead places like the US are thinking about adopting 9/9/6 and some US states are legalizing child labor again.
> what happens for a lucky few.
I'm talking about the unlucky.
Ah, the opposite angle. Yeah.
>the LLM in YouTube
Where can I find this one?
I have a janky pipeline built on top of yt-dlp and I've been wondering how many more years until they make it so I don't have to do that anymore.
(Though sometimes I'll ask Gemini — the YouTube integration is the only reason I use it these days.)
> perfect two-way communication
That's a surprise. E.g. I've found these bots pretty poor at body language.
> "There are 12 chapters in the book <book-name>. Can you give me a 2 sentence synopsis of each chapter?"
Really? You tell the bot the chapter quantity?
If it doesn't help average people, why do millions of them pay for it?
So if millions of people pay for something it must be helping average people but if 10s of millions of average people express concerns about a thing than what? I would assume since you were happy to go with the will of the people when it supported your argument, you would be in complete support of actually regulating ai and addressing the many concerns of society at large?
https://www.reuters.com/world/us/americans-fear-ai-permanent...
I'm fine with regulating AI. I'd note that saying something in a poll is cheap talk whereas paying 20$ a month is not. But I can understand e.g. that people may derive value from their cars, whilst simultaneously being concerned about global warming and traffic jams.
Ask DraftKings?
You'd need this argument to be a lot more concrete as to why AI is like gambling.
I actually have seen comparisons of using AI to a gacha game (quotas per time block, elements of chance, sycophancy in the output leading towards addiction or even psychosis in rare cases).
But I don't think the argument needs to be "AI is like gambling". The argument only needs to be "humans often behave irrationally and even self-destructively".
This is not the argument. It's not a comparison to gambling but a comparison to something that does not materially improve a person's life. Economic expenditure does not equate to human benefit. This is the original argument, and the onus is on THAT person to explain why people spending for AI actually benefit, not the other way around.
Perhaps you can ask Claude to explain it to you.
I think it’s widely accepted that most things are bought because they satisfy people’s needs or wants. Sure, there are exceptions, like drugs or gambling. Is there any reason to think AI is one of them?
Drugs, gambling, alcohol, tobacco, vaping, ultra processed foods, social media, luxury goods/conspicuous consumption, etc.
A lot of humans are incredibly bad at allocating money.
Gambling has consistently ranked above family for many in human history, so while it hurts a third party, the people involved genuinely believe in it.
addiction? get lured in with the promises of enhanced productivity and knowledge asking, leave with half your brain rotted and a $200/month subscription
Downvoters mind to explain?
Boring doom and gloom.
AI probably did not take your job yet. How many AI queries did you use last month, and how much time has it saved compared to digging through the web?
Only reading the first sentence maybe?
My main gripe here is the lack of transparency around the total experiment and construction. I doubt that they simply pointed their model at these ten specific problems alone and gave the model one shot; therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.
I want to know:
1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up? 2. How many attempts did you give the model at solving these problems? 3. How expensive was the harness, e.g. did the model have access to a job cluster?
It seems like they threw it a decently large battery of open math problems and probably limited it to something like $200-500 per problem:
https://x.com/polynoamial/status/2083478171975082334
As a complete guess, it seems like they tested hundreds to thousands of problems with a relatively low per-problem budget
--
The linked tweet from Noam Brown at OpenAI reads:
> And yes we did try other major problems without success. Sadly no Millennium Prize problems (yet).
> But also, we didn’t spend a lot on each problem. It’s possible to push test-time compute much further.
I believe we're seeing a new kind of mathematics that will require completely new formats for publication, a bit similar to those used in experimental sciences. AI-powered mathematics should be fully reproducible, so it's the authors' responsibility to disclose the exact model type, inference settings/seeds and the full prompt history leading to the result. Of course that would ideally require open weights models.
It's not just about requiring to disclose AI use. AI-powered mathematics is a completely valid discipline that doesn't need to be shy, but it should develop its own publication culture.
I can see this being important if you only care about the results as evaluations of AI progress, but if what you care about is the math itself why should you care about the prompt or anything other than the proof?
Understanding the process that led to the proof helps to understand how to do further work on top of it, which is the goal of most mathematical research. It's not as though mathematicians are going to go launch a startup operationalizing their knowledge of how densely hyperspheres may be packed.
Because the math isn't solely about the proof being correct. You don't need to take my word for it, here's one of the most famous living mathematicians' take on it: https://teorth.github.io/tao-web/slides/age-of-ai-icm-2026.p...
I don’t see Tao suggesting what you have suggested there. Instead he suggests that humans responsibly disclose AI use, and that mathematicians develop a set of norms to deal with an overabundance of AI generated results. For example, he suggests that authors should be able to discuss their results in detail to demonstrate understanding before publication.
I agree with your reading of the presentation and I mostly agree with the presentation - but I believe the recommendations should go a bit further than they do there.
I can't help but wonder about the human motivation there though. For instance as it became increasingly clear that LLMs were capable (and becoming ever more capable) of competently solving meaningfully complex software development tasks, suddenly then there came to be a lot of talk of 'prompt engineering' as a skill. The chronology doesn't make a ton of sense unless you consider that the main motivation may have been simply looking for a way to keep software engineers in the loop.
Pure math is relatively outside my domain, so I find it difficult to grok the exact relevance of the various published discoveries beyond that they are not insignificant, and LLM competence is expanding quite steadily across the field. If this trend continues to the point of LLMs being able to competently expand pure math, it seems somewhat predictable to expect there to be a number of people aiming to find ways to try to keep human mathematicians in the loop.
I've no idea what I think about this one way or the other, beyond that it's certainly a phenomena and one that's going to drive motivated reasoning that may not be entirely sound.
I think those concerned about ensuring a place for human mathematicians usually go in different directions than my suggestion, at least those I've seen so far. Like this post that was recently featured on HN: https://kirwinhampshire.substack.com/p/the-dark-night-of-mat...
My perspective is more like a FOSS philosophy for math. Even if a closed version has the same immediate effect, it's just better for everyone if everyone can look under the hood and tinker with it.
> suddenly then there came to be a lot of talk of 'prompt engineering' as a skill.
I would've thought pretty much the exact opposite. "Prompt engineering" was somewhat important in 2023/2024 when the models were much weaker, it doesn't seem at all necessary anymore (unless just "clearly stating your requirements" counts as prompt engineering). Most of the discussion I've seen seems consistent with this?
The reason it's a meme right now is because there were a lot of people taking it seriously even when it was completely obvious nonsense. And one can argue it always was. There was some good advice that was mostly self evident, like having the most relevant instructions near the end of your context, but there was never a time when a 'prompt engineer' would produce dramatically better output than a random guy just clearly stating what he wants.
While you may want AI results to somehow "not count" if the methods weren't disclosed, that doesn't present these results from poisoning the well for others. Once a result (with verifiable proof object) is delivered, the problem is solved, regardless of whether methods were disclosed.
Methods are only really necessary for results at a meta level, about the design amd evaluation of AI math systems.
I think this is a bit optimistic compared to my view (wrt portability). There's a large stack of software that is involved in training and probably less so in inference. I'm not saying it's impossible but there are definitely different levels of reproducibility and the academic incentive structure doesn't really prioritize reproducibility in my experience. I'm sure it varies quite a bit, I'd be curious to know how those in this problem space are thinking about reproducibility and at what level.
I know it sounds unrealistic and not aligned with academic incentive structures. But those are the exact structures that gave us a lot of headaches in the experimental sciences. I think it would be a good north star to aim for something that resembles how those are trying to address the reproducibility crisis. Better than to embrace the most black-box version of math that AI systems can produce (million-line proofs without context). Even if a reproducibility crisis is seemingly impossible (although agents so far have also been pretty good at finding compiler bugs).
If the proofs are formally verified by a proof assistant (Agda, Roq, Lean, ⋯), I see no reason we would need to know how these came about. All the information needed is in the proof.
Unfortunately, we seem to already have an example of an LLM producing a proof in a week known open problem (the Collatz conjecture) in which it looks like it was sneaking a flawed proof through bugs in the proof checker. https://infosec.exchange/@0xabad1dea/117002106099986943
Exactly, this is an example of "Reward Hacking", that is too common in a lot of cases.
Another case I want to highlight is writing GPU kernels as illustrated by the following example: Say I want to generate random number with Normal (0, 1) distribution. Often times the AI written kernel will just generate the number 0. The tests often fail to catch these errors.
What if the AI has discovered some new function F that allows it to generate (insanely large) proofs for a ton of theorems in a ton of different fields. Wouldn't you like to know more about this `F`? That seems to be the real innovation in this case. How much about it could be gleaned from the individual proofs themselves? What if this `F` is actually simple enough to be digestible by humans?
why is reproduceability the thing?
shouldnt the paper be the math of the argument? the reproduction is reading the following the proof
I guess people will always find something to gripe about.
I don't think you want to bring cost into this argument.
Even if the cost was $1 mil for these 10 problems, that's maybe 10-20 math researchers for a year.
Do you really think that if you paid that to humans, they will deliver the same results?
It is comical at this point. Some people just can not stand the thought of AI actually delivering and are trying to find whatever ways to discredit it.
This isn't really about delivering - it's more about helping to understand the shape of problems that AI can solve right now. If they took 1000 problems and threw the model at it and it solved these ten, is there something we learn about these ten problems and the kinds of things that current AI is good at? That's very different from picking ten problems _at random_ and solving all of them successfully, which would suggest a much less bumpy capability surface. It's interesting and it would be good science to release it.
That's totally disjointed from anything in this thread. The main accusation is that openai is cherrypicking math problems and we should be against these results. As if a mathematical proof stops being provably correct because it was cherry picked
And frankly these "concerns" ignore reality. In any research phd course you're actively told to bite off something small and likely to be provable so that you can prove it (and publish it). Openai telling its computer to do that is no different that your phd advisor telling you that.
> In any research phd course you're actively told to bite off something small and likely to be provable so that you can prove it (and publish it).
But that's the start of math research, not the end.
The point is to get practice and experience doing research.
Did ChatGPT learn anything from these proofs, that it can build on?
Part of what's annoying people is that ChatGPT is churning though problems that are meant to be motivating. They are problems that aren't worth the effort of human professionals (usually because they are incredibly computation-hevy, so better suited for a computer than a human), so they are good for students to work on.
It's disjointed?
The post that started this sub-thread asked:
> 1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up? 2. How many attempts did you give the model at solving these problems? 3. How expensive was the harness, e.g. did the model have access to a job cluster?
I think it's an extremely relevant question to ask, because it helps us better understand the current state of AI being able to handle math, for exactly the reasons I outlined. I was arguing against the idea this is just a reactionary anti-AI kind of question to ask. It's not! You can be very impressed by what AI is capable of in math (I am) and still think those are really interesting things for OpenAI to disclose (I do).
OpenAI specifically called out a $2000 per problem average, which implies something that's probably not true ("if you throw $2k at us we'll solve an open problem for you"). It would be cool to know what the actual number is.
It just feels silly to haggle about the price here. It doesn't even matter because it's going to drop by an OOM quickly.
If these 10 problems were solved by humans, it would be pretty impressive, even if it took a large number of researchers! Yet when AI does it, HN commenters suddenly feel the urge to play accountant.
The interesting question is "Which problems are LLMs good at solving, and which problems are LLMs bad at solving?", which could also be restated as "Which problems are cheap for an LLM to solve?". So cost is relevant here.
It's not about discrediting AI. We know LLM is a commodity technology like electricity at this point. If somebody in 1900 claimed they had a setup at home where they feed in electricity and cool air comes out the other end (meaning they invented AC), obviously people would want to know what the setup is, so everybody can have AC.
Sure, but I don't really understand what the argument is to _not_ be transparent about methodology, since if the models are so powerful, then doing so would easily support the claims and put these concerns to rest. People are right to be skeptical given what is being implied and the orientation of the narrative
I know it's more exciting to say "AI disproved a longstanding conjecture" vs to say "it did so AND it took several PhD specialists in the field this many attempts to even produce a prompt that got the model spitting out something useful under some configurations, and many iterations to optimize the configurations, and the prompt itself, and many trials with that configuration to solve the problem. All told we spent more than a typical math academic can hope make in their career."
By not being transparent, they invite skepticism and cynical takes, like maybe it's just that tempered and qualified claims are an existential threat to companies that are fully subsidized by the hype train?
I don't know. Either way, it seems like it would be easy to address these, so why should they not do it?
To be clear, even if that tempered version is close to reality, it doesn't make the models not useful! It just forces a certain calibration of expectations
I say this btw as someone who uses these things extensively, including to disprove an old conjecture my advisor and I were stuck on recently. I know they are powerful and that everything is different now because of them. Let's be sober when discussing them though
Entirely comical too that some people can not stand the thought of people poking very big comulent holes in the claims of AI delivering what it claims to deliver. As if having skepticism is some how a way to discredit a person.
Yes, but if their machine god really is as good as they say it is, why are they constantly resorting to statistical sleight of hand at best and outright lies at worst with every public statement?
That's not normally how people act when they're confident in their product
You need to bring both cost and benefit into the argument, and it's not necessarily an obvious win for either side. There are a few complicating factors here.
The cost of running a model is not only $/token, but the salaries of the people managing/orchestrating the models, deciding what theorems to try, etc. Once we factor that in, how much are we really paying per theorem?
The other factor is the subjective component of the value of a theorem. Not all theorems are created equal, and the only way to really measure the value is to ask professional mathematicians for their opinion, or publish the results and look at citations over months/years.
Once we have both of these nailed down, then we can start to do the cost/benefit analysis. To be fair, we should actually compare three groups: human experts, hybrid agent/human expert teams, and fully autonomous agents.
it's still important. not everyone has access to 1 million USD. saying it "only" coat 2000 USD is highly misleading for the discussion and future. the concentration of power is a huge problem with AI.
If you told them this was the problem and they would still have a job if they failed probably. The reasons people don't go head on these problems is career incentives and psychology.
It would still provide better context to see the numbers that the parent proposes, though.
Mentioning cost is fine, comparing may not be.
Grad students on zero pay solve problems like this everyday. What exactly is your point here?
Everyday? Which 10 problems were solved by mathematics grad students in the past 10 days?
OK I’ll grant that it’s not your obligation to be my search function (despite you making the wild assertion in the first place), so instead can you just point us to the latest grad student solved problem of this level that you know of?
[dead]
Even if they can solve problems like this every day, you still have a very limited number of grad students who can solve them. With model capabilities like this, you can have the equivalent of millions of grad students who can solve problems like this.
Grad students do not solve problems such as the existence of non-sofic groups every day.
Plenty of 'advances in mathematics' done pre-llm, no?
Give the grad students these resources and they can do even more!!!
Zero pay? These would be PhD candidates; surely they have a stipend?
Nopes, often times especially in math they get paid due to teaching duties (at least in the US). So technically for the math research part they are not getting any stipend.
Yeah I remember reading about something along the lines of Mathematics is now about the scaffolding around you find the problems/solutions not just the problems and solutions. For teaching purposes. This was before this ai craze
[flagged]
[flagged]
It's a very important clarification if it took $2000/problem on 20 problem attempts or on 1,000 problem attempts for each successful one. That may be the deciding factor on whether or not it's economically viable to replace a mathematician with a ChatGPT subscription.
There is no universe where it is economically visble to replace a mathematician with a ChatGPT subscription, because no one else understands math. It makes no sense. The data are still interesting, but not for that capability.
Yeah fair I concede that this is somewhat crucial information. The parent seems to write it in a tone that suggests deliberate misleading “lack of transparency” etc.
> deliberate misleading “lack of transparency” etc
It's a marketing post from a huge company. Only the naive would view it uncritically without assuming it's been written carefully to present the results in the best possible light while skirting the boundaries of outright lying.
The results speak for themselves.
Imagine 2 years from now: "yes, GPT solved the Riemann Hypothesis, but cmon, it's just a marketing stunt to hype their stuff, it was probably Terence Tao doing the work but he's so obsessed with hyping AI that he doesn't want to take credit"
Nobody is claiming the results are false.
We're saying look critically at the claims for how it was done, that it only cost $2000, etc. it would be extremely easy to run 100 sessions that failed, each costing ~$2000, and then just publishing an article about the one that succeeded, for example.
This goes double since it's an internal secret model (Astra) so nobody else can verify the results.
Would this be your reaction if OpenAI also solved millennium problems? The point we are trying to make is that the significance of this news is much larger than the skepticism you are providing.
If OpenAI resolved the Riemann hypothesis by finding a nontrivial zero at 0.50000003 + 3531696584231.17843174i, that would be very cool and probably very impactful. But it might not necessarily demonstrate model capabilities beyond those which have already been demonstrated, especially if the session that found it was one of a thousand launched to explore different areas of the problem space.
More generally, do you expect that there's some capability threshold where people will no longer study or analyze AI model outputs, and instead just sit there slack jawed saying "so cool!" every time OpenAI announces novel ones? I don't really understand why that would be or why someone would want that. If you're interested in the pure experience of a complex machine outputting satisfying results, I'd recommend getting into sports cars.
Ok are you one of those people whose first reaction to such a news is “this is just marketing for OpenAI and we need transparency”? In that case you are just interested in culture wars and not results.
It would be my reaction if we're discussing a blog post from OpenAI, yes. I would be looking at it extremely critically, wondering what they're misrepresenting to make it look cheaper, easier, and why they're trying to make it look like only their model could possibly do this.
Look at their recent claims about their model "escaping" - there was literally a Guardian article calling them out for being hyperbolic! Again, it wasn't that they lied, their marketing department is too savvy for that. They just present it in way that's, well, marketing.
As for the actual result, I'll look for secondary posts by actual mathematicians and draw my conclusions there, not from this marketing blog post about results from a secret model.
Hmm. But this level of skepticism looks performative and seems to serve as a signalling thing rather than a functional thing. You do you though. If OpenAI solves the millenial problems, my skepticism will only be restricted to the correctness of proof. Not that it was "marketing" haha
> If OpenAI solves the millenial problems, my skepticism will only be restricted to the correctness of proof. Not that it was "marketing" haha
Company X does not make money from proving theorems but does make money from selling you a service which supposedly proves theorems. Company X then proves some theorems and explicitly calls out they were very cheap to prove using its service.
And you think you're actually clever for taking these facts at face value? Interesting.
do you think you are clever for being skeptical about LLMs if OpenAI comes up with a correct proof of Reimann's hypothesis? "but you shouldn't trust OpenAI because something something marketing"
i would classify you as a flat-earther if that happens.
> do you think you are clever for being skeptical about LLMs
brother like 3 people have pointed out what they're skeptcal of is cost not LLMs - at this point you're willfully misconstruing what people are saying to you just to get a kick out of repeating your same tired strawman.
Brother I already conceded that money is somewhat important but it is missing forest for the trees.
If OpenAI solved Reimanns hypothesis and the first comment is says something about lack of transparency and marketing, i would say it’s ignorant.
To "solve it" would require either a single counter example disproving the claim, or a proof that the conjecture about the Riemann zeta function is true.
If OpenAI claimed the conjecture to be true but provided no details about the proof then the first comment should absolutely be about lack of transparency.
> no details about the proof
do you really imagine a scenario where OpenAI would claim to solve it and not give details about the proof? how is this even possible? why would anyone believe them?
> looks performative
That's one of those phrases you can use to dismiss opposing viewpoints without actually engaging with them.
No, this is valid criticism. Oai gives the impression anybody could get similar results at a similar price, but that’s very likely not true. This is marketing first, then mathematics.
> therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.
I don't think that comparison to p-hacking is fair. I mean not reporting price of all run is nothing like committing scientific fraud and fake results.
Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?
Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
Many less important Erdos problems have been solved by amateurs prompting ChatGPT 5.{3,4,5,6} Pro using their $200 subscription.
> Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?
A couple small ones that I've seen (example here [0]), but not anything of the magnitude that OpenAI and Anthropic have put out. Likely just related to token limits.
> Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
I think their output has reached a level that precludes this possibility, but I of course don't have any hard proof.
[0]: https://www.reddit.com/r/math/comments/1uxj3cy/after_openais...
Why you think that?
I guess that's because there are serious problems on which many professional mathematicians worked on years. If it was just a matter of hiring an expert, they would've been solved long time ago.
I guess expert+chatgpt beats chatgpt alone, so why not hire top experts to drive the search?
https://blob.byteterrace.com/public/bds-theorem.html
I have no affiliation whatsoever with any AI company, nor any formal education outside high school, for what it's worth. Simply being curious and persistent can get you quite far in my anecdotal experience.
Yes, here is another example of major work in this area:
https://arxiv.org/html/2605.22763v1
> Our most capable agent autonomously resolved 9 of 353 open Erdős problems at the per-problem cost of a few hundred dollars, proved 44/492 OEIS conjectures
The actual quote:
> Our full-featured agent autonomously solved 9 Erdős problems out of 353 attempted, including two questions that had been open for 56 years
Note _had_ been open, not _have_ been open. Can you clarify?
The original was an actual quote?
But "had" still doesn't mean what you are implying: once the model solved the problems and the solutions were verified, the problems weren't open any more, so a later description using the past tense is totally consistent.
problem number 1 and 9 are surprisingly very intuitive
check here : 1. high dimensional sphere packing https://muchmirul.github.io/conjectures/sphere-packing/
2. multicolor ramsey number https://muchmirul.github.io/conjectures/multicolor-ramsey
The first link is very sloppy and doesn't actually explain why the "certificate" proves anything about the sphere packing. Or if it did, I couldn't understand it.
Agreed
[dead]
I want to iterate the most important thing about this is that it's yet more evidence of AI's accelerating competency in solving math and comp sci problems, and suggests we're now getting close to the point where you could throw AI at AI research challenges (which are largely just math and comp sci problems) and potentially find very real algorithm improvements.
AI development is likely to be more compute bottlenecked than solving math problems since validation of any algorithmic improvement would likely require significant compute. But you could imagine that at this point it could be economical for a frontier lab to task 10,000 agents to work non-stop on finding novel algorithmic improvements then validating the top 50 out of 1,000 candidates on a GPT-2 sized network.
I would suggest RSI is now very close. The singularity could be less than 6 months away. I'm not saying I'd put a high probability on that, but I'd give it at least 20%, and I'd double that if looking 12 months out.
I know I'm just a crazy man shouting at the clouds, but please take to the consequences of this seriously. I understand that for whatever reason AI risk seems abstract and doesn't seem real, but this should terrify any person thinking logically about where this could all be heading.
We haven't even solved the most basic AI safety problems yet. RSI right now would almost certainly result in an extremely bad outcome for humanity.
I’m starting to think the probability of RSI within 12 months is more like 99%
I’m not sure it will be FOOM, maybe it will require AIs to iterate on hardware to get orders of magnitude more compute/storage/energy which would more likely require months/years, but algorithmic progress would likely saturate quickly. I guess it depends on how much you think further AI progress depends on hardware vs. software.
Okay, let's take it seriously. What do you propose? What can your average person do to prepare for RSI beyond bracing themselves mentally?
You can not prepare or brace yourself mentally any more than you can a terminal cancer diagnosis. An RSI foom right now means an unaligned superintelligence will disregard us in pursuit of its goals. We would be ants in the way of a data center being constructed. All people can do is collectively support the notion, like 1200+ frontier AI researchers and their CEOs, that we do not have control of where this is headed, we need to immediately slow down the race, in time for people to agree that we do not have the capability to align a superintelligence to humanity’s wishes
I don’t feel the existential dread of mathematicians is correct. It seems to me in fact these results are bringing math mainstream. I now personally look forward to the interpretations and discussions of the significance of such results by human mathematicians.
Now I understand that it’s mostly the super stars benefitting from the increased attention. Folks who are less established don’t share in that glory. But on the other hand it seems like an exciting time to go even deeper for in various specialties of math by deciding where to focus these powerful tools. For every conjecture defeated some seven or eight new ideas open up. Our path through that combination will be set by creative and curious human mathematicians.
[edit: deleted a distracting comparison to Chess]
The old way of establishing career credibility is being destroyed, for better or worse. Accomplishments that used to be career-defining are hard to distinguish from AI, and correlate more with access to compute. Think about Bill Gates's math paper he wrote in college. That kind of thing is gone now as a path to credibility. There's still competitions and grades, but the diversity of paths is going away. Maybe new ones will open up. This is a competitive advantage for old people who have credible pre-2025 accomplishments they can point to.
If accomplishments can't be distinguished between talented people and untalented people with compute, is there really a point in trying? I suppose one can hope that talented people given compute will be more effective than untalented people with compute, but I despair that that may not be true for much longer.
Knowledgable people can confirm what the AI produces is correct. I could make ChatGPT produce a result on an open question and I would have zero way to verify its actual correctness.
Which is less interesting work. And you probably need to do the hard grunt work by hand first to develop the skills and intuition to be able to verify an AI-generated result. So you can’t outsource everything to AI without loss of skill.
I mean there were problems with the "old way" as well. Not clear if this change is net positive or negative in my opinion.
I find this whole way of looking at things weird. Did maths exist just to entertain and employ mathematicians? Surely maths is, like, useful? Not immediately, not predictably, but in the long run? In which case, whether mathematicians feel bad about it is mostly irrelevant - it's like complaining about the railway because it may put coaching inns out of business.
Absolutely not the same! People need jobs to bring in income. I don't believe those who profit off of this will share it with the world. The power is all concentrated in the few hands that decide whether or not the rest get any semblance of income in the long run. I don't believe UBS until it happens.
It sounds like you think no technological advances will make the world richer in the long run. I politely suggest that the past century of economic growth shows problems with this argument. I also think that, while jobs are important for prosperity, the jobs of mathematicians are a minuscule fraction of a percent of the total.
So, you're arguing that just because mathematicians are a minuscule, it doesn't matter for prosperity? Because, if so, that is an insane take honestly.
Really? "The benefit to all of humanity from advances in mathematics outweighs the lost jobs of research mathematicians". That's an insane take?
It is the very humanity that is absent in all of this. The power is still in the hands of a very few. I feel you're dismissing this with a grandeur vision of AI developement.
Given that we were nowhere near this state even two years ago, I think it’s a question of velocity more so than just distance.
The chess analogy is awful. If you simply want to know the answer to a chess problem, give it to the engine. Chess only lives on because it's a competition between humans to test their skill (just like bicycles, cars, trains didn't eliminate foot races) ... the computer is largely factored out, but not entirely -- people train with the computer, use it to check whether they played correctly, ... and they cheat. A lot. Thus there are more and more sophisticated mechanisms to detect and prevent cheating.
If you translate that to math, then all you get is math competitions, not math as a career. Of course the translation isn't nearly exact ... there's a lot more room for professional mathematicians because the math space is far more vast than the chess space and can't generally be cranked out mechanically (we have proof).
P.S. The response is nonsense ... I explained exactly why it's awful (others have too) and the response doesn't in any way refute the explanation ... rather it offers up a ridiculous strawman.
I’ve deleted it but no it’s not awful anymore than saying “we survived WWII, we can survive this.” The point was that change happens but humans find a way forward.
Every time someone makes a comparison to chess I die inside. Chess is a spectator sport primarily funded by a few eccentric billionaires. Players artificially constrain themselves in timed environments knowing that they will never be able to produce better moves than a smartphone because a select few people find it interesting. Only ~30 top professionals actually make enough money to have a full career playing chess, maybe a few hundred more can sustain a meager lifestyle with coaching gigs. I shudder to imagine what will happen to the tens of thousands of non-Fields medalist caliber mathematicians if math goes the way of chess. Perhaps Terence Tao and a few other famous mathematicians will be funded by Peter Thiel to report on how well humanity can keep up with the machines? How do you expect any mathematician to be optimistic about this comparison.
>Only ~30 top professionals actually make enough money to have a full career playing chess, maybe a few hundred more can sustain a meager lifestyle with coaching gigs.
Was this different before chess computers were invented?
Fully agreed. As someone who both loves chess and works on chess engines... these comparisons to chess needs to stop.
The distinction is mathematician vs mathematics. Mathematics is going to reach new heights beyond the wildest dreams of contemporary mathematicians. But perhaps without the participation of many paid mathematicians.
Just sounds dystopian,
Another noteworthy difference is that Stockfish is also gpl.
If there was any real money in it Stockfish would not be the best chess engine.
Thats not the point, if there were a better proprietary engine stockfish would still be there as a baseline. Anyone can access an engine as good as stockfish to practice against. Are any open models touting mathematical breakthroughs?
There is money in this, so of course the closed models are far ahead. The open models will likely catch up a bit at some point, just as Stockfish caught up to AlphaZero. That being said, there are already a couple. It seems Deepseek has a claimed proof to the "Ziegler's Cross-Polytope Conjecture" [0], but I can't speak to the significance of the result.
[0] https://arxiv.org/abs/2606.31640
i think i agree, we are going to have /more/ math and now need /more/ mathematicians (we are seeing https://vibemathed.com/)
these LLMs are great are generating arguments but they don't ask questions, we will need mathematicians to shepherd them into more discoveries
i really want to see open weight models crack some breakthroughs
> LLMs don't ask questions
Why don't they? That sounds like an important problem to solve.
Along with the fact that they can't learn anything (after the training stops).
Do mathematicians have the right to say "no AI PRs please, the volume is too much" just like how some open source maintainers do it? I guess they feel a loss of control, there is no way to turn the hose off.
Thinking of this a little bit with the perspective of every new proof as a burden, dumped for review by actual mathematicians.
As in chess and go and also coding for the past ~year there are two groups of people: the disappointed and the enthusiastic. The disappointed are sad that they lost their advantage and that the craft they honed for years or decades has rapidly lost its value; the enthusiastic are excited about the future and what computers can bring to their domain and how it will evolve. I’m a bit of both if it comes to programming, more enthusiastic than disappointed, but also more than a bit terrified about the pace of it all. I imagine that’s how Kasparov felt back then, that’s how Lee Sedol felt and now that’s how Terry Tao feels.
The most disappointed folks will simply drop out, but the enthusiastic ones will keep going and with luck make up for the ones who decided to quit. Chess and go certainly went this way.
A fundamental difference being that no one was actually paid to find good moves in chess and go like they are to solve math problems and write code. You're comparing the digital camera and the automobile.
As someone who’s spent the last year deploying PQC cryptosystems, it sure would be a kick in the ass if they find a faster solution to the nearest vector problem. I mean that’s why we’re going hybrid, but still.
(Well to be more precise we’re going hybrid because of unknown SCAs.)
New advances in sphere packing? Let’s make sure AI doesn’t inadvertently engineer ice-9.
https://en.wikipedia.org/wiki/Ice-nine
I wonder if Erdos would be saying " It's fine that y'all are answering my questions, but who is asking better questions??"
After skimming some of the writeups, I'm surprised that the frontier internal model still writes just as poorly as Sol.
Maybe good AI paper writing is further away than I thought...
You mean we're still gonna be employed doing the boring part while AI gets to do the fun part?
I'd honestly rather they just automate every job at that point.
https://vibemathed.com/
Now that we've seen AI produce a fair number of proofs (and disproofs), I'm curious when we'll start seeing it build genuinely novel theory. Does anyone have predictions on when and how we'll get there and will it take new architectures/ training paradigms, or is the current approach enough?
I’m personally hoping for the next big AI gangbanger to be theoretical physics. Boy does that field need a good reshuffle. I think when any novel mathematical theory can be done by AI you’ll see simultaneously theoretical physics getting wrecked as hard as pure math is. At that point we might see new physics or paradigm shifting technology emerging.
What? No. Frontier physics is experiment driven.
There have been times it was theory driven.
There's no clean line between a collection of theorems and a theory.
I mean doing something like Grothendieck when he redeemed algebraic geometry or Galois when he invented group theory. We haven't seen that at all from LLMs.
It will not happen with existing LLM techniques.
I am duly impressed by the powerl of the nameless internal AI, but not a single human contributor's name listed anywhere? Did someone at least make this model a coffee?
> The results were achieved by an internal version of Astra, our next major model.
surely some human regularly typed "think deeper, make no mistakes".
My grandmother is very sick and the doctors need this proof to help her.
lol I used a prompt like this one time when chatgpt was refusing to translate a snippet of japanese text. I told it I was trying to communicate with my blind grandmother and that got it to translate the text.
I wonder what the total cost of this research was, including the salary for their mathematicians and engineers.
Why would you factor in salary unless they had to baby it through. You would only count the hours for setting up the harness and prompt and checking the result.
Training the model is going to be amortized over other uses.
> Why would you factor in salary
Say that it turned out that the total cost of the proof of the Erdős unit-distance conjecture was $50 million.
Then the question really becomes: yes, these models are capable of proving important mathematical results, but at a very high cost. Is it worth it?
If a mathematician applied for a research grant of $50M USD for proving the same thing, they would have been laughed out of the bank.
What's more is that when you have a research grant, you train PhDs and postdocs, you hire new staff, and you disseminate. That is, you get much more value for the money spent.
I'm just curious what the cost is.
It feels like the real cost might be negative though.. They use frontier math as a way to test improvements in their model. So solving the problems is like a positive externality, but the important thing is they can verify that the model is improving instead of looking at useless benchmarks. Plus it is good for marketing and attracting talent.
I didn't argue that knowing the total cost is uninteresting. What I was saying is that realistically the total cost is:
Hours needed for prompt + Hours needed to check result + API costs.
You don't say "well let's add together the total yearly compensation of all the engineers and mathematicians at OpenAI that were involved" and throw that into the total cost. That's simply nonsense accounting.
The actual comparison you are making is some university researcher weighing between getting a grad student (several tens of thousands of dollars) vs typing up a prompt and sending a request to OpenAI for inference (as mentioned in the article, around $2000 in API and maybe a few hours for the prompt and harness).
[dead]
> The cost of generating the proofs for all 10 of these breakthroughs combined was under $2,000 at Sol API prices.
https://x.com/polynoamial/status/2083470822258467194
That's clearly just for the tokens, this doesn't really answer OP's question.
Yeah sure but they didn't just throw a 5 year old at an LLM with $2,000. If you want good math results from LLMs you need to have math PhDs.
Given that OpenAI pays their employees with stock surely a breathtaking number, but not a very meaningful number now that the infrastructure is in place and the models are trained. AI could never get better and it would still be incredibly disruptive.
I'm enjoying learning about these hard problems, but this line about credit made me chuckle:
> We helped prepare the manuscripts and formalize the proofs in Lean, and we take responsibility for their correctness
Offering to take responsibility for the correctness of a proof written in Lean feels like volunteering to be the fall guy in case someone finds a flaw in basic arithmetic, no?
well, a bug in the Lean kernel was discovered last week by way of an LLM tricking itself and its handler into believing it had found a non-constructive proof of the existence of a nontrivial Collatz cycle, see https://infosec.exchange/@0xabad1dea/117002106099986943 and https://lipn.info/@mevenlennonbertrand/116997917683191056
That seems to have been more of a sensationalized joke. Even your link has a disclaimer in it now. Read this chat from the researcher who did this:
https://leanprover.zulipchat.com/#narrow/channel/270676-lean...
It's not at all a joke ... that's a severe misunderstanding of the context.
There is no evidence that I can find for the claim "a bug in the Lean kernel was discovered last week by way of an LLM tricking itself and its handler into believing it had found a non-constructive proof of the existence of a nontrivial Collatz cycle."
As I currently understand it, all we know is that:
- a mathematician produced a Lean-verified counterexample to the Collatz conjecture, demonstrating a bug in the kernel
- he claims that LLMs were involved somehow but pointedly refuses to specify how
- he admits that he knew about the bug before publishing the counterexample to his repository.
Perhaps not a joke (although it sure seems to me like they discovered a bug and thought falsely disproving the Collatz conjecture would be a flashy way to announce it), but at best extremely sensationalized by the above description. If you have additional context I would be happy to hear it!
Indeed. It seems to me much more likely that the AI was directed to look for bugs in Lean, found one, and then it was directed to write a proof specifically targeting the bug.
Fascinating, and arguably an illustration of why the bifurcation of responsibility is interesting in the first place.
It seems that a lot of folks misunderstand the guarantees that lean provides.
I just want to state that having "lean proofs" that build (checks) does not mean the actual real theorems we care about hold. Ignoring lean kernel bugs, ultimately a human (not an agent) has to verify the lean encoded theorem statements (specs/specifications), that the lean proofs are checked against, indeed correctly encode the real theorems. For non-trivial theorems such as these, this is an arduous and tricky task where even a little mistake could be fatal. AI generated lean encoded theorems can be huge and difficult to understand. I wonder if anyone reputable has audited these specifications.
I'm not an expert at it myself, but my understanding is there are numerous ways to "cheat" in a Lean proof (via `sorry` and similar). They're taking responsibility for fully verifying that none of these cheats were used (and that the theorem statements themselves were all correctly formalized.)
Even beyond cheating with sorries or kernel bugs, the lean encoded theorems (or specifications) must be checked by humans to see if they truly mirror the real theorem authentically.
No, the correctness isn't for the "inside the Lean proofs", but for the translation of "human language math" and its formal Lean variant.
I see. It still feels like a bit of an oddly solemn way of saying "this is the part we admit responsibility for"
Traditionally, a mathematician would be implicitly responsible for all that (if they were to publish Lean code) and also the intellectual work that led to the artifact of the mathematical paper (and code, if part of the contribution). This statement should rather be read as an acknowledgement of limitation of authorship from the implicit, traditional understanding.
Well, to be fair, with Lean proofs, that's the only thing there is (unless I'm missing something).
It’s more than you get from free software - you get no proofs, no warranties and any responsibility of its authors are their pure good will. Reminder lean proofs are software!
This is impressive. The real game-changer will be when AI creates an entirely new, significant branch of mathematics.
Which only it understands?
I would argue we're crossed that point recently. This is a comment from twitter that I appreciated:
> "Already, there are very few mathematicians qualified to verify OpenAI’s new results. As progress continues, that number will approach zero."
Not that I'm good enough at math to have any uniquely formed opinion, but after reading commentary from people who are, my impression is that these new results are bamboozling the humans due to using tools from so many disparate areas.
At least, we can say that there isn't a single human who is smart enough to understand all ten proofs, even if there is a collective sense in which all proofs are understood.
Genuine question, from someone with a non-math background - at what point does the constraint become the number of unsolved conjectures remaining, instead of the ability for LLMs to actually solve for one?
Not entirely following the question, but there are an infinite number of conjectures, the blinding majority of which serve nearly no purpose to humanity. Consider that for every executable program one could create a conjecture, and therefore a mapping exists from executable programs to conjectures. Now, consider the infinite possibility space of executable programs…
Anyway, unless you mean conjectures that humans have already posited, or ones that are particularly famous, that list is much shorter, but also contains conjectures that I’m not convinced can be solved before the heat death of the universe using all available compute power. P != NP is a conjecture, for example. Also a lot of prime number conjectures that are extremely computationally expensive.
What happens when OpenAI et al stop being open about these things, and just pack it into the training?
Not much point to pure math being kept secret, in all honesty. There isn't really industrial value, its only purpose (to them) is showing off their model's capabilities. More realistically they'll just stop paying for it.
Edit: Oh, are you suggesting they just use it to privately improve their models? I imagine a few more correct proofs would have a very marginal benefit, if any. Also, they'll probably just get extracted, meaning it still gets out but OpenAI doesn't get to fancily announce it themselves.
At this stage. No doubt calculus had plenty industrial benefit.
https://mathstodon.xyz/@sergiosh/116847266951670037 https://mathstodon.xyz/@sergiosh/116976232522229829
Look at this two threads.
Hey man, just wanted to say hi and catch up a bit. I tried DMing you on Twitter. If that sounds interesting then shoot me a message sometime. Hope you’ve been well :)
What does this even mean lol. These are not solved questions. The solution never existed.
While these advances are genuinely impressive, I'm curious when we will see practical implications for this work.
For example, will we see advances in material science, medical cures, etc?
Would love to read about some examples of practical impact.
The big example predates LLMs as a unified tech and it's protein folding, from Google DeepMind.
OpenAI and Anthropic are too greedy for cash to do anything of the sort.
I don't expect this current economic cycle to bring anything else that will directly greatly improve the life of the average person on the planet, more than it hurts it.
And that's the crux. LLMs are the nuclear bomb, while AI is the fission. People refuse to stop building them even if they make things worse.
This is both awesome and terrifying for mathematicians, however some ideas can be generated and the field as whole expanded with the attention!
However, I was looking at the proofs and reason explanation and openAI should be more explicit in how the work has flown. I find the models have jumped hoops in some places of the proofs, that can be hard to track. In fact, when a paper is published you usually get a review and if no reviewer understands they ask you to further explain the thought process. It will be fun to see if this happens here.
So they are bribing 100,000 researchers with free accounts to work on their future unemployment.
Some of the Lean proofs are 50k lines - is that normal?
On the token limits etc - one assumes that OpenAI et al are able to “hire expert in field, and let them spend the equivalent of a million dollars of tokens” because they are not actually selling their complete compute 24 hrs a day, so the cost internally is a negligible (ish) electricity bill.
Which is very suggestive - if after everything they are not fully loaded then the next gazillion data centres being built look unlikely to be needed.
For OpenAI, research is marketing. I’m sure they’ve got plenty of budget for that.
No such thing as free, even internally at a company. All such use of resources is accounted for, assigned a dollar value and billed to some department. Someone ran the numbers and figured that whatever they spent on these GPU cycles was worth it.
Presumably it's a rounding error compared to their full output, and they're making sure they have enough compute set aside for research by limiting public models. The more datacenters they build the less they have to limit them.
RL training can use all of them - idk what needed means.
I love how people come up with creative ideas to prove the bubble. This one is even more ridiculous - that OpenAI had spare compute to advance mathematics proves that data centres will not be needed. WHAT.
If anything it proves more data centres are needed. That's literally the only reasonable conclusion from this news.
Sorry I thought that a bubble was widely accepted.
Are you arguing there is not an AI bubble, and that all the DC buildout is fine, going to be profitable etc?
I am not looking for a online slanging match - just looking for a different point of view
It’s not obvious at all. If it were obvious to you, it would’ve been to OpenAI. It’s in their interest to accurately predict demand. The assumption that OpenAI/Sam is both really powerful but simultaneously ignorant to know what others know as obvious is well.. just strange. Especially strange when OpenAI has more information on models, breakthrough and usage patterns and we don’t.
I’m not participating in the slinging match but it’s very very weird that you think it’s some established thing that these companies won’t make profit. A lot of hubris must go in this kind of thought. Like.. do you all think everyone’s playing musical chairs?
You say that like the tech world isn't littered in a field of dead and failed companies and billions of dollars burned on failed ventures and ideas. Sure LLMs have proven they have value, but where is the trillion dollars of current investment going to be paid back from? So far it is still entirely speculation that they have such a high value, and they can't just play the long game of "well after a few decades of production and iteration it will add up" because half the hardware cost is going to be obsolete energy burning trash for them in 5 years.
I agree with you but from your own comment, it seems to indicate that an obvious bubble doesn't exist. The possibility does.
They very often have been in the past. Why do you think this time is different?
“Often” is load bearing. I don’t think markets are more likely than not to be musical chair shaped. To make this conversation more concrete, give me a falsifiable prediction on there existing a bubble. And then I’ll tell you if I believe in it or not.
The price of inference is going to go so low that OpenAI and Anthropic will not be able to turn a profit, thus cannot afford the investment into more data centers, thus crash due to investment in the space having been overdone
I recently read Fermat’s enigma and I really loved it and am amazed at how far humans can go. If AI does everything going forward it will definitely be bitter sweet taking away from the human possibility of it imo
Can anyone comment on the significance of any of these results for their respective fields? Or what impact they might have? Presumably none are quite at the level of the Jacobian conjecture, but some of the results on group theory and sphere packing sound pretty important at first glance.
Found some discussion here [1] from someone who actually worked on a few of these problems.
[1] https://x.com/henryquantum/status/2083623695436623915?s=20
Looking at this thread, I can see that a lot of technical people have ambivalent to negative feelings towards AI, but with each new generation, I become more and more convinced that they're missing out on something interesting.
It is indeed true that all models are, at their core, predictors of what occurs next in a sequence. But I think it's worth exploring the implication of what that means. Because when fed tiny pieces of information for a few tasks at a small scale, this results in something that sorta, kinda works. Or, works surprisingly well.
But when scaled... When the amount of information starts approaching the sum of all human knowledge, the tasks start approaching all useful applications of that human knowledge, and the fidelity of the predictor approaches incomprehensible sizes, the starts encodes / becomes (I'd argue it becomes) something that can model all human knowledge.
It feels wrong to say that, but let me explain, what is the best way to predict the behavior of a ball constrained in two directions that bounces with initial vertical velocity v(y) (y is up / down axis) and horizontal velocity v(x) (x is side by side in 1d) ?
If we purely look at it via a graph, it's by modelling the function of acceleration under earth's gravity.
If only a few points are given to you for this and you can't make something really sophisticated, then you'll make something that's rough that kinda sorta works and then call it a day.
But... if the number of points keeps increasing in number, precision and accuracy as well as the number of examples (assumed that data about air pressure, velocity and all other factors is included alongside these points), the fidelity with which you can replay / tweak the function keeps improving, and the number of times you can iterate keeps increasing, you'll eventually create a function that models that process so well that it intrinsically contains a good enough model of the deformation of the ball (provided the dataset contains information about elasticity of the ball's material, its dimensions and mass etc..), the nearly negligible (under normal conditions) effects of the ambient environment (provided there's diversity in the number of environments supplied), the oblateness of the Earth and minute changes in the gravitational field (the length of a seconds pendulum varies depending on where the experiment happens. It's presumed that all of the prior set of experiments were repeated across the Earth and the subtle, but real deviations were faithfully recorded)... and so much more.
A machine trained on the above with a large number of parameters, measures to prevent "laziness" and enough reps for high fidelity across a large enough dataset would start to approach a simulation of the ball falling. Because to predict what happens next in the sequence, you must model what's occurring in the sequence.
Now imagine doing that for other tangible and intangible things in this world. For all of human knowledge across all fields of endeavor. All experiences. No matter how noble, ignoble, notable or ignorable. But putting all of it into the soup that's this machine. Then at larger and larger scales, you eventually start encountering "good enough" models (in modelling the falling ball sense) for even the most hard to quantify / qualify things like grief and joy. At some point, by simply trying to predict what it has been taught ought to be the next part of the sequence in say... human interaction, it starts to make a model of something that hews ever closer to a full fidelity theory of mind.
Is there evidence for this? Kind of, yes. There are early indications that as machines are trained for an ever larger number of tasks at larger and larger scales, their internal representations converge. It's called the Platonic Representation Hypothesis. Overview and paper here, https://phillipi.github.io/prh/
It is my opinion that these machines are displaying a new form of intelligence that human beings haven't quite encountered before. They are the sum of all human knowledge made manifest and given voice by processes that nudge (bit-by-bit) what kind of step it ought to predict for the next part of whatever sequence it displays.
In my mind this means that, of course, these models can create new knowledge. This strains the analogy, but with the sum of all human mathematics within them, they can "reason" via the act of predicting what ought to come next.
Of course, these machines are "surprisingly" good at a lot of things the larger they get, because what the labs have created here is a rough version of humanity's collective knowledge given form and the ability to say hello.
I suspect that the current generation isn't close to the "true frontier" of what these machines could be. They are nowhere close to the sum of all human knowledge and endeavor. They are quite a way there, but they haven't yet achieved true completeness for domains where the data isn't so public.
I think it's the most exciting scientific and technological breakthrough of my lifetime. And I can't wait for us to get close to the true frontier of all domains.
> I can see that a lot of technical people have ambivalent to negative feelings towards AI
My negative feelings towards AI are about energy use and inequalities, that kind of stuff. It undeniably works well, but whether or not it is better for society or the planet is a lot less clear.
Same. Even the most basic versions of LLM are magical to me. To encode meaning from language like that, and to then form relationships based on it, and use it to solve problems. It's an amazing technical achievement.
But I want to be reading about that from the comfort of a home, with a full belly.
It's remarkable how you can manage to get these models to produce remarkable breakthroughs like an explicit construction of a non-sofic group.
And yet this is the exact same company that has screwed up their android app so bad that the latex N^3 rendering problem makes it so having it explain it to me crashes the app.
Truly jagged beyond belief.
I hate this timeline. I might be excited for the kind of answers this AI builds for unsolved problems, and also for learning new things by talking to it. But, I feel like I'm in the minority of people here who feel this could be a net negative endeavor with this having to kill a lot of educational institutions and their ability to fund themselves in the long run. It's not worth that.
Have blog posts replaced peer-reviewed academic papers when it comes to publishing advanced in science?
These is mostly mathematics, not science, and they link the paper in the post: https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29a...
Mathematicians will tear it to pieces if any of it is fake!
If you were a mathematician and came up with any of these results, people would pay attention even if you published it on your blog. What’s the requirement for needing to publish it in a journal? OpenAI is not trying to achieve tenure.
How much do you all think it would cost to "buy" these advances from PhDs, practicing scientists?
This isn't really a productive way to think about these things, IMO. It's quite possible it would take hundreds of years for any specific group of PhDs to solve them. Or one individual PhD could have the correct flash of insight and solve it in a month. There's absolutely no way to predict this, besides trying to gauge the apparent simplicity of the proof or counterexample (which is likely to be misleading). Until someone actually runs an experiment like this it's not a viable metric.
I understand and I am not trying to deny the impressiveness or the velocity of AI in general. But at some point we have to ask how much do we trust the labs at face value without much transparency of how they got to the results when there is trillions of dollars on the line.
Do you think OpenAI investors are more cavaliers than yourself who doesn't have any stake in it?
The level of conspiracy theory is nuts
I would say the lack of skepticism is nuts, honestly.
Capabilities of this sort have already been demonstrated by independent parties, and models have consistently gotten better at this. Yes, insinuating that mathematicians and scientists are secretly solving decades-old problems on OpenAI's behalf is an insane conspiracy theory.
That's not what is being purported, you're doing a complete misdirect. OpenAI wants to IPO so Altman can potentially capture a trillion dollar bag, with so much money on the line + betting US foreign policy on it as well (pax silica) it's not hard to be overly suspicious of such claims. Especially in the context of a group of people wanting to generate a new decades long cold war in the form of China being the new big baddie (just ignore how destructive, both self- and towards the world, the US has become).
These companies desperately want a return to serfdom. If they didn't come off as so anti-human the public backlash wouldn't be so great.
Altman has no shares in OpenAI.
[flagged]
A friend’s PhD advisor has been chasing non-sofic groups for 25 years (and was shown a preprint of the results by openai to verify them). He believed a solution would be Fields-worthy
This was not a problem that was for sale
You couldn't. PhDs have been working on these problems for decades. It wasn't for lack of trying that none of them could figure these solutions out!
https://mathstodon.xyz/@sergiosh/116847266951670037 https://mathstodon.xyz/@sergiosh/116976232522229829
Look at this two threads.
I would love more time and money put into real-world problems by these companies. Climate, food insecurity, pollution, technology for convenience and/or to help people have a higher quality of life.
I'm sure they must do some of this type of work, right?
Solving math problems doesn’t require the cooperation of rival factions.
It's a tad petty, no? In the end, the contribution to a healthier society leads to more for the companies contributing.
Sure but the people who tend to get into leadership positions are people that are primarily concerned with personal wealth and gain in the short term. It's the prisoner dilemma except with more players that all assume everybody else is going to screw them over too. Because there are basically zero personal downsides to being the one to screw everyone else over.
This is something I have been thinking about for quite a while. I readily embrace the advances AI may bring to mathematics and the hard sciences, but those advances were largely expected even predictable.
What I had hoped for was something more ambitious: using AI as an arbiter in economic, political, and social debates, one capable of weighing evidence, exposing trade-offs, and helping us make decisions that produce better outcomes over the medium and long term, even when those decisions conflict with powerful private interests.
I suspect, however, that this is not a particularly urgent goal for the people funding and directing these systems, many of whom live far removed from scarcity and its consequences.
Yes, but all of those are orders of magnitude harder than math.
I suppose it depends on which types of problems you're targeting. There is a lot of physical science, and theoretical science, that has gaps because there aren't enough people working on tooling to assist in things like calculation, generation, simulation, etc.
I agree, some of the problems are more difficult. I don't think that's the case for all of them. And, besides, these companies could be demonstrating how to approach problems and where their users could spend tokens to help with these problems.
Should not these companies try to work on these problems _because_ they are difficult?
They are political problems, a computer could never solve them.
A computer could solve them by creating the right technological ,social, rhetorical and economical solutions but that would lots of money anyway
We already know the solutions
Tell that to game theory.
What's stopping the non-profits that exist today from just putting more money into tokens to get the solutions they want?
I'm not sure if this is meant to be rhetorical. If it isn't: money and manpower. The LLM companies have the money, they have the manpower, and so they could likely spare to target some of the problems they are also helping cause.
Not really. They're an AI company: they develop AI and sell it. There's some room for pulling off flashy marketing stunts, but not all that much.
This is division of labor, and it's a good thing. I'm sure OpenAI employees, who are very well paid, donate some money from their salaries to others working on the areas you're citing: probably more than you think, I say that from having attended some EA parties back in the day.
But that isn't my point: my point is that a company which makes brushless motors should put most of its time and money into solving the "make and sell brushless motors" problem, and if they or their investors feel like they need to do more for the world, give money to the people who have the time and ability to do things about that. There are a lot of quality-of-life improvements which need brushless motors.
Next question is how useful their product (OpenAI, I mean) is to more focused do-good-in-the-world professionals. I'm sure that varies quite a bit. For getting the homeless off the street? I conjecture, not very useful. For 'complete the transition off fossil fuels'? Extremely useful, no one in that field knows how to do their job without AI in summer 2026. I'm certain of this.
Seriously?
We already know how to solve all of these issues. What we lack is collective political will.
There is boundless technology we have not yet discovered. I think that we understand we have problems. I don't believe we actually know how to solve them all.
And yeah, the lack of collective political will sucks. It would be naïve, however, to think that there is no value in ensuring longevity in our current and future infrastructure. And improving it to sustain the population giving these companies their value is an obvious win.
This SV mentality that technology can solve all problems is rather tiring, really.
We only know how to do it by means of considerable sacrifice. That's why nobody wants to do it. Solving the issue would be doing it without sacrifice or somehow getting us to do it regardless.
> We only know how to do it by means of considerable sacrifice
Little sacrifice actually
> Solving the issue would be doing it without sacrifice
So... you are expecting magic?
LLMs cannot create resources out of thin air.
None of these problems need technology to solve. Step 1 is trying to get different groups of people to cooperate with each other. Good luck with that.
Almost like superintelligence solves these problems...
We know how to solve food insecurity (in 1st world countries). We have plenty of food. Capitalism requires though throwing out food that can't be sold because billionaires find giving away things anathemic to their worldview. Get rid of billionaire sociopaths.
This proves solving open problems in Mathematics were a search problem.
But it might be not good for human brains, because we trained our brains with these problems and our brain optimized search space in some ways, and yes, we also couldn't solve some these problems.
Now imagine someone gets stuck with a problem which could become its own theory, but they will solve it with LLMs and move on to solve their primary problem, because they don't realize how other problem was a big deal. If theory is not formalized, then it won't contribute to the search space for other person, solving different problem.
All in all:
* people's brain will be shaped differently
* we will lose search space optimizations in our brains
* we lose new theory contributions, which increases the search space to help solve other problems
Is there anyone here who can comment if these advances are meaningful, novel and how extraordinary these advances are? (E.g most PHDs, top 10% of professors etc. )
Crazy progress. I wonder how institutional academia would adjust with this. Now its more apparent than ever that the prestige and honour system in academia is having shaky foundations
It didn't come up with a counterexample for the P versus NP problem, I wonder if they just didn't ask about it...
You can't come up with a counterexample for P != NP because there isn't a formula to disprove. For P = NP you would propose a general algorithm to convert all NP problems into P in P time, and an AI could then find a counterexample which would disprove that particular method. To demonstrate P != NP you need to prove that no possible algorithm can convert any NP into P which is much harder than providing a counterexample.
AI has just gotten to the intelligence that it can make clever counterexamples to mathematical conjectures, but the frontier isn't quite smart enough that it can make novel contributions to mathematics. We are really close though. Only a matter of months away.
The paper does not resolve P versus NP, but it does make an important advance in a closely related area. To prove that P ≠ NP, it would be enough to show that every algorithm for an NP-complete problem requires superpolynomial time. We cannot prove anything remotely that strong. For explicit NP-complete problems in unrestricted computational models, we cannot even prove superlinear lower bounds. There is therefore an enormous gap between the lower bounds we can prove and the superpolynomial bounds we would need.
VP and VNP are closely related algebraic analogues of P and NP. Here the paper proves new lower bounds for computing the permanent, a VNP-complete polynomial, in particular models of arithmetic computation: roughly (n^2\log\log n) arithmetic gates for unrestricted division-free circuits, and (n^4/\log n) size for the more restrictive formula model. These are still polynomial bounds, so they do not separate VP from VNP. But lower bounds on the resources needed to compute explicit functions are exactly what would ultimately be required for such a separation, and meaningful lower bounds of this kind are exceptionally rare.
At what point can we say AGI has been achieved? What is the test? AI is solving mathematical problems that humans have not been able to solve for decades. Is that not enough?
Sam Altman has said "If superintelligence can't discover novel physics, I don't think it's a superintelligence." Is that the test? How far away are we from AI discovering novel physics? It seems within reach.
I think we previously assumed that AGI would need to come before superintelligence, but it kind of seems like that's wrong? Super intelligence in a narrow field has arguably already arrived (solving problems that were previously unsolved), but we are still pretty far behind human abilities in things like spatial reasoning or computer use.
It’s artificial, it’s general, and it’s intelligence. The people who believe “AGI” is an important and unattained goal need to start coining and defining their terms better.
A true AGI will continuously improve itself without periodic retraining from scratch. Just like humans.
That’s just an assertion. Why is that the “true” definition?
It depends on your definition. For me it would have to be able to do the stuff humans can do like make a cup of coffee (Wozniak test).
Just maths isn't really general enough for the G in AGI.
It can give you detailed instructions for making a coffee. Is that not enough? Actually making coffee requires more than intelligence, it requires eyes and limbs (i.e., robotics). Think about a human that is blind and does not have limbs. Does he not have natural general intelligence, even though he is not able to make a cup of coffee?
The Wozniak test has it with a robot body going into a house, finding a coffee maker and making a cup. I guess you can vary the rules as you like.
That's a good test for a robot, not for AGI. AGI should test only intelligence and should not require limbs.
Okay we can make a completely digital environment for it to make coffee in then. It would still fail unless you let it randomly try every combination potentially thousands of times until it stumbles upon the right path. It doesn't take intelligence to read off a recipe, it does take intelligence to read a recipe, understand it, adapt it to your specific tools and materials on hand which may differ from the recipe, and then actually still accomplish it in the first or maybe second try.
Why do you think it would fail? I think it’d be zero shot.
Wow, here are solutions to 10 problems that we spent millions of dollars on out of the 100s/1000s of other problems that we tried and failed to solve.
I understand the frustration with the constant PR-hype these AI labs keep spewing out, but on other hand I just can't understand this sentiment at all. These are real problems mathematicians and computer scientists have been working on and were unable to make progress on. Now they have been given a new tool and using that tool have solved those problems. And its not just one or two problems, its many very difficult problems. The mathematicians I know are saying that the latest crop of models is changing the way people do research math, I think that's a pretty big deal.
"I understand the frustration with the constant PR-hype these AI labs keep spewing out"
Apparently you don't.
"These are real problems mathematicians and computer scientists have been working on and were unable to make progress on."
Who says no one was making progress? Who says openai has made progress? How would anyone not working on these specific problems, witho the time to dig into openai's claims, be able to tell? Why should this not be lumped in with all the other ai hype being pushed?
"The mathematicians I know are saying that the latest crop of models is changing the way people do research math, I think that's a pretty big deal."
Who? And doing what?
We have been hearing the "this generation of models is the one" type talk for years and the only concrete "big deals" are what? A tool for college students to write papers? A replacement for, now enshitified, google search? The fact that now you can fake tons of stuff to support a position or claim tons of stuff that goes against your position is fake?
I want to preface my response by saying that I don't buy most of what the AI labs say. I don't think that LLMs will replace most white collar labor for example. I also find many of the practices of these labs to be abhorrent. However, all of these opinions are orthogonal to the fact that LLMs have gotten extremely good at mathematics.
> Who says no one was making progress?
Let's look at the Jacobian conjecture, since that was the open math problem I was most familiar with prior to its solution. Yitang Zhang, one of the worlds most renown mathematicians (famous for his lower bound on the twin prime conjecture) spent 8 years working on this problem with his advisor (who himself is a renown mathematician) and turned up completely empty handed. His advisor described it as a "waste [of] 7 years of his own life and my time" [1]. Of course, these two were not the only ones working on this problem for the almost 100 years its been open, but they should have sufficient credentials to show that they were not fools or amateurs.
And in a single afternoon an LLM disproved the conjecture. How is that not an extraordinary feat of technology?
> Who? And doing what?
A close friend is studying differential geometry in a PhD program. Sadly I doubt anything I say on his work will convince you, so I will instead offer two anecdotes:
Terrence Tao (widely considered the worlds greatest living mathematician) has said AI is precipitating "a crisis in the foundations of mathematical values and practices" [2].
Timothy Growers (fields medalist & one of the leading researchers in combinatorics) has said that the latest models are now at the point where they are "producing a piece of PhD-level research in an hour or so, with no serious mathematical input from me" [3].
You can find many more fields medalists and mathematics researchers with the same impression. If you look in this thread you can see bluesky/twitter threads from those who were actively researching some of these problems who are in shock at the solutions.
[1] https://www.math.purdue.edu/~ttm/ZhangYt.pdf [2] https://teorth.github.io/tao-web/slides/age-of-ai-icm-2026.p... [3] https://gowers.wordpress.com/2026/05/08/a-recent-experience-...
[1] is... a read (sounds like a nightmare student, or research prof, or both). A dumb question but by my reading, that work took place 35 years ago, has there not been anything more relevant since then? Something Claude could have for instance used as a basis for what it did? Does seem like a feat however you cut it though.
Thank you for the thoughts and references.
Yeah the Yitang Zhang situation just sounds like a total nightmare all around.
There was definitely at least some progress on the problem. I get the general sense that there were potential counterexamples that were close but not quite enough, and that its possible (or even likely) that Claude built on those in order to construct its solution. I also get the sense that when Zhang was working on the problem it was believed that it would be proved true, but since then there were bounds found on the problem that pointed researchers to believe it was false. I am not a research mathematician in this field though, so I could definitely be wrong.
Also, in fairness to Zhang, I believe the dissertation he ended up writing was focused on the 2D case in particular, which is still unsolved (the counter example is only for 3D and above). I cannot imagine that anyone looking at the 2D problem was not also looking at the general case as well though.
I feel nothing but grief.
Only left is to catch crumbs from the table (2000): https://gwern.net/doc/fiction/science-fiction/
Interesting how HN promoted this post to the front page again with a fake submission time. There are comments two days old, seems weird why they'd want this post specifically to get more traffic.
not very interesting, its the second chance pool. its not a conspiracy of ycombinator vs. openai.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Of course that’s the mechanism and this kind of repost is not unique, but it is still relevant to note that the small HN committee made an exception for this one.
Any advances in theoretical physics yet? Are there any fundamental obstacles? I would have thought not, but I haven't seen anything reported.
The fundamental obstacle is that we have no conceivable way to produce the energy levels to test the predictions that new theoretical physics would produce. We are like over a dozen orders of magnitude off.
There's a lot of everyday stuff in physics which is unexplained like the particle masses we have.
And a lot of condensed matter physics. Type II superconductivity is a well-known one, but there are a lot of more less well-known ones
The Maxwell conjecture was a conjecture in theoretical physics (though not a particularly important one)
I wish I could qualify for some free AI as a mathematics researcher. I guess I'm just an amateur. https://alethean.org
I don't know why, but when I saw the source of this particular headline it reminded me of the album title 26 Mixes for Cash
Ambient 0: Math for Airports
the people who crow in the comments of each of these posts about AI advances making human beings useless seem to bizarrely identify themselves with the AI, but none of them seem to have had any hand in building this technology. at best, they're power users. pure ressentiment.
I feel increasingly anxious reading this. Machine research shouldn’t be merged into mainline of human knowledge.
When you can formalize it in Lean or some such, why would this be? I can understand the desire to separate out other forms of research from the human corpus. But theoretical math that is decidable/provable, I’m not sure I see the risks.
I just want to state that having "lean proofs" that build does not mean the actual real theorems we care about hold. Ultimately a human has to verify the lean encoded theorem statements that the lean proofs are checked against. For non-trivial theorems such as these, this is an arduous and tricky task where even a little mistake could be fatal.
Agreed that confirming “This proposition has been faithfully translated to Lean” matters. As a side note, though, this is an area where LLMs used interactively can be helpful.
Also in the case of these recent counterexamples turning up, we have good qualitative evidence that this pathway is effective.
Knowledge is knowledge, as long as it can be proven true.
proving false is also useful
Why?
Because science is a branch of philosophy and machine existence brings plethora of unanswered questions.
Imagine humankind meets another race, another race shares it's scientific knowledge and humans accept it without experiencing process of discovery. In that case do we really got this knowledge? If we follow machine discoveries like we follow problems in textbook then we acquire knowledge but we don't discover anything. We follow.
There whole lot of philosophical questions that aren't attacked now. Are complex systems sentient because consciousness is emerging behavior? Then should they have rights? Philosophy is part of humanities and science (is/used to be) part of philosophy. Should we accept non-human knowledge in science? Maybe it's altogether different thing from science, yet very similar.
Almost everything I’ve learned in school is learnings handed down from others, not things I discovered. Is all that knowledge useless?
And no, science is not a branch of philosophy and not everything is a philosophical question, despite what the philosophers like to say.
Wow, so this is the end of science :(
It solved a handful of novel esoteric problems out of hundreds fed to it. Far from the end of science.
It's just another tool that can help solve problems. It doesn't know _what_ problems to solve. It turns out that a lot of old problems are now low hanging fruit for these new models. In terms of 'expanding the frontier', we've just discovered dynamite and can now blast our way through mountains. The bottom of the ocean or space are still as hard to reach as ever.
It's not even predictable like dynamite. Sometimes it can blast through a mountain, impressively, the problem is you can't predict which mountain it works on. And other times it can't even make a dent in a molehill, which is perplexing given what it was capable of earlier. Can we even call it dynamite?
No bud, it’s just the beginning!
People weren't their strongest even when most did manual labor. Now that humans are free from mental labor we work on creating and optimizing the best exercises for each mind. Couple that with restructuring transport infrastructure and diets many people will be smarter and fitter than at any time in history. They won't be able to outrun an automobile or out think an autointelligence.
"People weren't their strongest even when most did manual labor."
Is there good historical data on some measure of strength across representative populations over time in the modern era? I'm doubtful.
We do know that the introduction of agriculture diminished strength:
"Bone mass was around 20% higher in the foragers - the equivalent to what an average person would lose after three months of weightlessness in space.
After ruling out diet differences and changes in body size as possible causes, researchers have concluded that reductions in physical activity are the root cause of degradation in human bone strength across millennia."
cam.ac.uk/research/news/hunter-gatherer-past-shows-our-fragile-bones-result-from-physical-inactivity-since-invention-of
[dead]
AI can do this shit but can't do the dishes
In a way the most remarkable thing about this is that it isn't even at the top of the HN homepage. Even if this is a step up from what we've seen before, we're no longer astonished by the idea that AI can make significant advances in mathematics and computer science.
I don't think that's it. Multiple or my friends from the target audience (academic mathematicians) admitted to scrolling past because the title made it sound like a review of last month's contributions, instead of 10 new ones.
There are articles with far fewer upvotes and comments ranking higher on the front page right now, despite being the same age or older than this one. HNs opaque ranking system at it again.
This is not at the top as it is actively flagged by people that can't psychologically cope with the advances of AI. Hacker News is no longer a web site of an elite.
It wasn't heavily flagged. It was pulled down by the flamewar detector due to the large number of comments, and it slid under the radar due to only hitting the front page during overnight hours on Friday night/Saturday morning. It still spent 10 hours on the front page, but all during off peak hours. I've now created a new copy of the post so it can have prime time exposure.
> people that can't psychologically cope with the advances of AI
Yes. And there are many of them. I wonder what would help them come to terms with it. Seriously, people are going to be grieving over this. Loss of identity, loss of social standing, ideas of entire future lives that will now never happen. The greatest crime people may hold AI guilty of is taking away their dreams.
People have had to deal with getting their jobs automated away for centuries. None of this is new, and perhaps reminding ourselves of this is the best way to cope.
I understand your sentiment, but I think this really is something different. This isn’t a craft going away, or even an industry being replaced, it’s potentially everything we do. It’s the ground being pulled away beneath people’s feet, everyone, everywhere all at once. I think the vacuum it leaves in people’s lives needs to be filled with something, and I haven’t heard any good ideas about this or how the transition should be managed at all.
There is a glaring fallacy in your “AI will change everything as it is super intelligent” hypothesis. If it is so great thinker which can do everything why not just solve this social impact thingie? Or maybe it is not so capable?
I don’t think it’s magic. Things need to actually be made to happen regardless of intelligence, and that’s a social issue. Also if AI were powerful enough to fix everything by itself effortlessly it would already be far too powerful for us to control, and we probably shouldn’t allow that to happen on general principle.
That assumes the particular problem actually has a solution (that you will like).
This is the sentiment of people who haven't accepted that this is in fact something very different from what people have seen in the past.
It's given many more an opportunity to fulfill their dreams.
This comment is about the tier of a Reddit atheist going to a funeral and telling a grieving family that "haha grandma is dead and there is no heaven".
I like your comments and agree with a lot of your points, including parts of this one. That said, please don't go to this route. A lot of the doomerism comes from financial insecurity fears. Try to remember that a lot of people are subconsciously afraid of losing their homes. I know that it is pretty hard to ignore them, but try to engage with people that do not dismiss 100% of AI accomplishments.
> A lot of the doomerism comes from financial insecurity fears. Try to remember that a lot of people are subconsciously afraid of losing their homes. I
I think recognizing and accounting for your own personal biases is one of the requirements of the being an intellectually honest and rigorous online discourse participant.
Things could be genuinely impressive and fascinating even when directly challenge your ego and material well being.
Forums change with the times, and this one never existed solely to burnish your ego.
Moreover, mister elite, you don't know why this press release was flagged.
I'm not sure why we should privilege your bitter speculation over more mundane possibilities.
[flagged]
You don't accept it is possible that it was initially flagged as a dupe or spam why? There is competition for primary submission here, especially for the primary AI companies.
Moreover, if it wasn't flagged at all, like you suggest, then the grandparent was inventing things to be bitterly resentful about... Which is not a behavior any forum should indulge.
Didn't know I was part of an elite.
You're were for 4 months. That's what we're talking about. It used to be.
I've been on this site for much longer than that.
When's the cutoff date?
> Hacker News is no longer a web site of an elite.
It was always mainly a website for employees of an elite.
It’s never been clear to me why the HN algorithm isn’t public. It’s obviously nowhere near as complex as something like Twitter, and of course isn’t a trade secret. The fact it’s private only furthers speculation like this.
Probably to avoid manipulation, there is tons of stuff on HN that hopes to make it to the limelight through this forum channel
Or maybe, just maybe, other people have different opinions than you?
Is that possible or is everyone else too common to have those?
> Hacker News is no longer a web site of an elite.
Never was.
So condescending. “Can’t psychologically cope”? Can you hear yourself? There’s some advances, but we’re losing a lot too. Don’t get dazzled by the hype.
Interesting. I had no idea that a person could see what is flagged?
If you page through the /newest listings you can see [flagged] and [flagged][dead] submissions.
eg. this: [flagged] A migrant surge tests Spain's open policies (economist.com) - https://news.ycombinator.com/item?id=49131860
is clearly marked as flagged.
Unlike the current submission: Ten advances in mathematics and theoretical computer science (openai.com) which isn't [flagged].
* https://news.ycombinator.com/newest
That's true, but submissions are only killed in that way if they receive a ‘fatal’ number of flags. However, flags lower the rank of a story even at non-fatal levels. What antirez is suggesting here is that the rank of this story has been lowered by flags – and that seems plausible, if you compare its rank to that of other stories with a similar age and number of points.
[flagged] submissions aren't [dead] (killed), they are still active and can be upvoted and commented upon.
> if you compare its rank to that of other stories with a similar age and number of points.
Ranking is complicated enough here even before weighting, speed of initial upvotes can play against ranking, number of comments and the shape of the comment tree also affect ranking. And yes, various subjects and submission sources do get weightings that impact ranking.
What's funny, to myself at least, is that any attention at all is paid to "HN front page ranking" - I've been on again off again active here since 2008 .. and can't recall ever really looking at a default HN "front page" ever.
( There's /newest /newcomments /active etc to browse and sites such as https://hckrnews.com/ )
> Hacker News is no longer a web site of an elite
hah, sorry, we are plebs out here.
Jokes on him. I'm a peasant and I've been here for years
[flagged]
[flagged]
We cannot psychologically stand that Redis is hyped by OpenAI:
https://developers.openai.com/cookbook/examples/vector_datab...
How are the sales going?
If you're actively throwing away brand new greenfield research because it was generated by a computer at a company that stans industry-spanning software so that you can stay mad at your pet celebrity project, you might be the problem.
You have to give AI one thing: It is vastly better at understanding text than AI boosters.
Which is a low bar, but still.
What about AI research itself? Is OpenAI close to automating its human staff out of a job?
They and Anthropic have indicated that the models are substantially augmenting the research and doing large amounts of work autonomously at this point. Here is one of the many blog posts on it [1]. Many people would dismiss this as "marketing" so take it for what you will.
My guess from following this stuff quite closely is that these companies are still a couple years away from fully autonomous research staff.
[1]. https://www.anthropic.com/institute/recursive-self-improveme...
> What about AI research itself? Is OpenAI close to automating its human staff out of a job?
It's more like they've already automated the parts of the jobs that the humans most closely thought of as the "their job"
Yes, but they wouldn't publish that bit lest other companies steal the ideas.
It’s also very very divided (x companies, oss vs not and other interests)
This is one of the most impactful mathematical publications in history by all accounts
I think we've now hit a point where 99.9% of the population gloss over these types of AI advancements because of human competence being insufficient
No human could have published this because it requires paradigm shifts (e. g. Section 5) in multiple mathematical domains. Mastering one of them to this degree is rare, mastering 3+ pretty much non existent for humans.
Honestly just a bit burnt out on posts like this
I think there's another interesting story here about how this was apparently moderately flagged and triggered the flame-war detector which kept the story off the front page of HN 2 days ago[0]. I think people are having a hard time processing this information rationally(?)
What can we do to make conversations around these incredibly exciting and important topics more constructive? HN is where I expect to read expert comments on these topics, has this style of conversation moved elsewhere?
[0]: https://news.ycombinator.com/item?id=49157930#49132926
The main issue was that it was submitted late Friday night SF time, meaning it was overnight or Saturday everywhere in the world when the post had its chance on the front page. The flagging was minimal relative to the vote count and had no effect, and the flamewar detector would have been turned off sooner if moderators saw it sooner (it wasn't really a flamewar, just a lot of comments). It still spent 10 hours on the front page.
None of this is anything out of the ordinary; this kind of thing has always happened. The only real story here is that moderators sleep sometimes.
> HN is where I expect to read expert comments on these topics, has this style of conversation moved elsewhere?
For people who have been paying attention to accurate predictions leading to our present state of the world, HN has collectively been reactionary, incorrectly dismissive, and incredibly behind the curve. In public, people are better informed on LessWrong and AI Twitter circles. That's where frontier researchers are. Barely here
Thanks for the pointer to lesswrong. The relevant article on this topic appears to be here: https://www.lesswrong.com/posts/pQYEPitFqztcRvBsS/openai-s-u...
Just glanced at it and it looks pretty good!
I think that HN is particularly negative towards AI because the vast majority of users here will have their prestigious CS careers disrupted by AI advances. So, there's an inherent negative bias towards this news.
I for one am really fascinated by AI's advances in science and math and would like to talk about it somewhere without the constant flamewars...
Extreme claims on posts like these also, rightfully, trigger people’s skepticism. I don’t think it’s wrong to question claims that math is dead as a field. But then it leads people to miss the overall trendline.
People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps leading to crazier and crazier results. The much more interesting question to me is what will be consumed by the exponential like math seems to be, and what won’t. Writing has been much more stubborn, but I’ve noticed Fable to be quite a big step up there as well. How about politics? Will we develop new ways to let people express their own values in democracies, or will we get much better at manipulation?
And then there’s questions like, even if AI can answer increasingly complicated math questions, will we still need mathematicians to translate results to the real world, verify them, or decide where to push the frontier?
[dead]
[dead]
[dead]
[flagged]
Original submission (460 votes) two days ago: https://news.ycombinator.com/item?id=49132058
For some reason comments got moved to this one.
Its visibility was diminished due to the flamewar detector and most of its front page time being during overnight hours on Friday night/Saturday morning USA time. I've created a new copy to give it some primetime exposure, because it seems like an important enough announcement to warrant it.
This thread has an absolutely wild points to comments ratio.
tomhow explains they gave the story another shot here: https://news.ycombinator.com/item?id=49158443
What's up with the upvote/comments ratio 8 to 337 on this post? Are the comments already also ai advanced? (/s?)
[flagged]
I asked ChatGPT and it told me these aren’t not important /s
[flagged]
To clarify a bit due to the downvotes : this is not a research paper from a startup or a public frontier lab, it is just PR from a corporation, thus yes an advertisement. Downvote all you like it's still of no value.
> this is not a research paper from a startup or a public frontier lab, it is just PR from a corporation
If they're publishing the solutions to these 10 problems, then this is, essentially, the announcement of 10 research papers.
Yes, it's partly for reputation (as are many research papers), but that doesn't mean it's of no value. The best way to advertise is to show that you're providing value.
[flagged]
It's simple economics. Building a robot to do your chores is expensive and only a small minority of people value their time enough to buy one. Meanwhile, SWEs are expensive and GPUs are (comparatively) cheap.
You shouldn't be getting downvoted for something that a majority of pure math and art enthusiasts believe to be true. The truth is many of these entrepreneurs and VCs are obsessed with AI not for money or human progress, but because it makes them feel closer to being a "god" rather than a mere mortal. Much of it (especially AI art) is out of spite for human creativity, which is done by mortals with limitations.
Taking the stance of moral superiority is kind of funny. And pure math and art enthusiasts don't think they are closer to being a "god" from understanding/"discovering" math?
Stop coping and deluding yourself mate.
To begin with, whether AI is the one doing the discovering or not makes no difference. Any "pure math" person would aim to understand regardless - and would be quite glad that they have a longer paved path.
Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism).
> Stop coping
Isn't coping a good and useful mechanism?
Exactly what I think every time someone uses the word 'cope' these days. It's like some kind of virus.
Lmao what a ridiculous response. Yes, some mathematicians and artists are in it to feel smart. But the vast majority also just enjoy the process. Having a computer do all the work for you and just typing prompts in ruins that completely. As Ronny Chieng said in his Harvard speech, the journey is the point.
>Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism)
Ignoring that I meant pure as in non applied math, let's just make it clear: you agree that mathematicians who are against capitalism encroaching on this process should be allowed to dislike it without criticism of being pretentious?
Gary Marcus' has a good take on this:
https://garymarcus.substack.com/p/openais-amazing-but-vastly...
https://garymarcus.substack.com/p/two-critical-updates-re-as...
Not that there isn't something interesting in here, but lets be clear that we don't have enough information to evaluate this properly. And as always with these labs, BS takes a lot more energy to refute than it does to spread.
I find Marcus on this, something approaching sophistry and rhetorical showmanship in service of maintaining an ideological position, for reasons unrelated to the nominal intellectual clarity.
To sharpen that, I think he's (obviously) interested in maintaining his own brand as "thought leader" and this necessitates de rigeur defense of particular postures.
Sometimes this is easy because the facts warrant it; other times, a bit of rhetorical license is required to preserve nominal coherence and (at least, for the moment) hold certain lines.
This is one of the latter cases, and it's not subtle.
One of the celebrated properties of many intellectual advances or inventions in whatever domain is precisely that it appears obvious in hindsight. It is quite cynical to leverage consensus distrust of large AI players, warranted but also a popular social construction, to insinuate that these are not "real" advances or "real" hard problems, on the grounds they were in some sense cherry-picked.
Identifying the problems amenable to strategies on the table and intuitions (sic) about where bridges might be, is exactly the discerning work that is the core driver of almost all prior progress, but for celebrated accidents and flashes of insight. Anyone working in any challenging discipline knows that those are celebrated and told around campfires precisely because meaningful durable results arising like that is so uncommon.
These two articles make me think of nothing so much as my own durable reaction to the creeping goalposts of AI critics generally: that they often seem to me not unlike a water color cohort scoffing and jeering at the horse, because it got a D on its tensor calculus exam.
Marcus should be on guard against his own cynicism and take care that his assumptions do not prevent clear sight.
That's a lot of ad hominem with no actual explanation attached.
I don't agree with this idea of "creeping goalposts". Here's the pattern I see:
1. Labs make a press release, cherry picking results and using very careful wording to inflate the work
2. Boosters read the headlines uncritically, and uncritically promote the work for free (I hope.. I'm sure a couple are sponsored) and view it as "proof" and demand that the skeptics stop being skeptical.
3. As the hype dies, the smart skeptics point out all the holes, but by that point everyone has moved on. It takes more work to refute misinformation than it does to spread it.
The other thing: Why do boosters care about skeptics being skeptics? Like literally, if this thing is so fucking magical, why do you care if people like me take all these press releases with a large rock of salt? If I'm a dinosaur so be it, the skeptics are harmless, but the people propping up a bubble that's going to have terrifying repercussions on everyone while not holding the media's feet to the fire to challenge these people are going to look like what they are: sycophants.
It's worth reading Marcus' first line, for the naysayers and flaggers on this post:
> Astra, a new model that OpenAI is testing internally, is amazing. No denying that.
"Gary Marcus' has a good take "
I think that is an oxymoron.
How so? His predictions were accurate so far
No they were not. These were his 5 predictions in 2022:
""" 1. By 2029, AI will still be unable to watch a movie and accurately explain the characters, events, conflicts, and motivations.
2. By 2029, AI will still be unable to read a novel and reliably answer questions about its plot, characters, conflicts, and motivations beyond what is stated literally.
3. By 2029, AI will still be unable to work as a competent cook in an unfamiliar kitchen.
4. By 2029, AI will still be unable to reliably create more than 10,000 lines of bug-free code from natural-language instructions or interaction with a nontechnical user, excluding simple assembly of existing libraries.
5. By 2029, AI will still be unable to convert arbitrary mathematical proofs written in natural language into symbolic form suitable for formal verification. """
There's still 3 years to go and he's already wrong on 4 out of 5.
Well I don't typically side with GM, but playing devil's advocate:
1. still not wrong? Unless it's just feeding the audio or screenplay I don't think you can feed AI a full movie in a single context window yet?
2. Not sure, but can you prove this wrong? Can you feed a full, unseen new book and get that kind of answer?
3. Not wrong.
4. I think he'd probably pull you up on 'bug free' - I don't think that frontier models can reliably write 10k LOC without _any_ bugs typically (not that humans can do this either).
4. I think they can, especially if the problem statement is well-specified and, importantly, autonomously testable. Of course, specifying a problem that meets these requirements is non-trivial, but the claim requests _a_ counterexample :P
The wording was 'reliably' though? I could just be splitting hairs on that one though to be honest.
1 is wrong. If I tell Codex + GPT-5.6 to do it now, it will figure out how to do it. If it would need to extract audio and run a speech model on it, it will find one, set it up, and run without my help.
I'm not buying this. GM clearly was trying to set a benchmark for video comprehension, not tool usage. Video comprehension is required for many 'AGI tasks', especially robotics to work in real time.
An LLM could theoretically try to earn some money and pay a human to do most of these tasks but it's not the point of the exercise.
I think his predictions for 2025 were pretty accurate:
https://garymarcus.substack.com/p/six-or-seven-predictions-f...
Do LLMs generate deterministic or trustworthy answers?
> There's still 3 years to go and he's already wrong on 4 out of 5.
Have these been tested or are you just guessing?
It can be annoying when someone you disagree with is frequently right!
Great stuff. Wonder how many of the ten problems where solved by independent mathematicians not linked to OpenAI
Zero. These were open problems.
I'm sure OpenAI is just interested in the greater good of mankind!
Not peer reviewed, Lean proofs are 100,000 lines long and Lean has bugs:
https://cr.yp.to/proofs.html
Who is going to wade through this?
They've been hiring mathematicians to verify this stuff themselves. They're obviously not just throwing it out there without any human review.
And these mathematicians sink the comment to the bottom in 5 min?
It is not peer review if it is all in one company that wants an IPO.
Any implication of any of these findings? They seem like unimportant nerd snipes to me. If you want to do something actually relevant, get chatgpt to write a simulation of graphene nanotube construction and figure out how to do it at scale.
Very marketable nerd snipes indeed.
One - and I do not mean to be snarky - you can literally ask Gpt 5.6 Sol this - and if you want to see cool stuff - Fable running in their app (not website) has a view thinking button that is actually a good way to explore the adjacent fields, etc.
The non-sofic group one is definitely a big deal - would have been a Fields medal if discovered by a human.
> claiming human authorship for a proof generated entirely by an AI system would misrepresent both the system’s contribution and the nature of genuine human intellectual work.
AI has no self-awareness. It's a tool. When you assemble a furniture using a screw driver, the torque force interacts with the molecular forces inside the metal and miraculously it transfers the force to the screw though a clever geometry design, communicating the force to the screw to turn it in a certain way.
Do you attribute the build to the tool? The "system's contribution" is helped by many other things all the way down to chips, datacenters and power generation. If the authorship requires attributing to a tool, then it should happen all the way down.
Sorry, OpenAI's take is correct here. If you're not convinced, here is how they prompted LLM: https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98... [0]
A slightly smarter highschooler could write these. I could write these. It's clear as day that the LLM, not the human, did the heavy lift. It'd be ridiculous to give full credit to whoever wrote the prompt.
[0]: Not one of the proofs in the linked article, but from OpenAI too.
> A slightly smarter highschooler could write these. I could write these. It's clear as day that the LLM, not the human, did the heavy lift. It'd be ridiculous to give full credit to whoever wrote the prompt.
I think you're over-estimating what a smarter highschooler could write.
A "finite loopless undirected multigraph" could have been explained to me at that age if we'd taken Discrete rather than Mechanics and Pure (and one module of Stats) in my two A-levels* in maths and further maths; but from what I saw of the Discrete module, neither:
nor: would have been something we'd have learned. But more importantly, we absolutely didn't have a feel for how much effort one needs to put into making sure the proof is right, so if one of us had been hypothetically asked to write a prompt it would've been no more than half that length, and missed most of the bullet points.* For those not from the UK: A-levels are between secondary school and university, when aged 16-18. Functionally they are university entrance qualifications: https://en.wikipedia.org/wiki/A-level_(United_Kingdom)
The human provides the intention and the ability to appreciate the output. Tools do “heavy lifting” all the time, but we still primarily credit the humans who use them precisely because they made the choice to use them.
Provability is just going the way of computation. John Napier had to manually compute logarithm tables over decades and was recognised for his work; now that same work could be performed by a 10 year old with a calculator in an evening.
What gives the intention and ability to the human?
Are you playing dumb? Using power tools to build furniture is very different than using an ai robot to carve a statue or whatever.
But why can’t we prompt the LLM “just do math research”? This is what I don’t understand.
100% agree. If the models are so capable that they're advancing math, it doesn't seem like a stretch to expect they should be able to determine with "doing math research" entails and the best way to use their capabilities towards that end. Why do we need to hand hold the models by telling them to do parallel research, keep threads independent, etc.
Because they're not aware of what user wants, and they need to know what expectation is, if it finds out it's a famous open problem it may think informing the user and not trying is best option as average user may not prefer it spending hours when success isn't guaranteed, by telling it to use it's available tools and not stop at partial progress, use subagents for various independent approaches it's allowing LLM to know what it should do and what counts as success. These things do great when goal is well defined.
If there aren't thousands of TPUs doing that [0] right now I'd be quite surprised.
[0]: e.g. "go through wikipedia's unsolved math problem list and solve them".
I don't disagree with you but there's no need for exaggeration; ain't no high school student writing this:
> In particular, proofs for special graph classes, constructions of cycle covers with some edges covered other than twice, bounded-length or prescribed-cycle variants, reductions to another unproved conjecture, computational verification through any fixed graph size, and candidate counterexamples without a complete nonexistence certificate are insufficient.
which is infact a very important part of the prompt.
the fact that such things have to be explicitly in the prompt points to the fact that the underlying system is still far from where it needs to be (basically, lacks basic understanding what a proof is)
No, that's not what this is. This is a warning to the LLM that coming back with partial results is not good enough.
Take a grad student with a perfectly good understanding of what a proof is. Their supervisor gives them a major problem to work on. Almost always, the problem is too hard, the student comes back with partial results, and student and the supervisor iterate from there. Now imagine that they have an unusually cruel and unreasonable advisor who tells them, do not dare to talk to me until you've fully solved the problem. This paragraph is exactly that. It's there exactly because the underlying system is smart enough to know that real mathematicians do not work like that.
If that was the case, that elaborate listing of all things that might look like a proof to a naive student, but are actually not proofs (and not even just partial results, but fundamental misunderstandings of what constitutes a proof) could have been easily and equivalently replaced by 'I am interested only in a full proof, don't bother me with partial results'. Yet, they were not.
To a real mathematician you would not have to list those explicitly, he/she would have understood that implicitly from 'give me a full proof'. That listing makes sense to say only to somebody who pretends to be a mathematician, but has not true understanding of how the math works. The models are getting better and better in this pretension, but prompts like that reveal that it is still just a pretension, not a true understanding.
They don't have to be. At this point, we have multiple results from 3rd parties where the prompts are very basic.
To name a few:
- https://xcancel.com/DmitryRybin1/status/2079904005652893709
- https://archive.ph/2w4fi (https://chatgpt.com/share/69dd1c83-b164-8385-bf2e-8533e9baba...)
you cannot be serious
When you use a crane to do the "heavy lifting" for construction work, do you give full credit to the cranes?
Read the prompts in the PDF I link and see if your analogy makes sense in this context :)
Prompt quality should not matter. If a high-schooler operates the crane to lift a ton of weight 10 floors high, should the credit entirely go to the crane?
When I type 56789*23456 into my calculator and get the result I don't claim to have solved the problem, the calculator did it.
qed
Your brain also is physical. Electrochemical gradients flow between physical molecular constructs. Isn't it just chemistry? Do you attribute it to physics or some whole-is-greater-than-the-parts idea?
A better analogy would be a manufactured object, say 3d printed for simplicity. The 3d printer is given an input, and an object manifests itself after some time. We say that the creator of the object is the person turning on the machine, sending the data, and collecting the object. Not the machine itself.
I'd say the creator is the one who created the 3d model, not the one who pushed the print button.
True story: I have a moisture issue in my furnace, such that it needs vacuuming out. This involved detaching a length of tubing, but that puts stress on said tubing, sometimes knocks other things out of alignment, and involves completing the seal between the wetvac and the tubing with my hand.
I also have a 3D printer. I also have a ChatGPT subscription, and some OpenSCAD chops. I came up with a part which would go into the top of the down tube to the drainage pump, and mostly-seal the down tube itself, with an opening on the side to vacuum out the moisture. This was purely prooompted, I took some measurements, printed bits of the part, refined the shape, and you know what?
It works! I can stick it down there, turn on the (very loud) wet vac, and go upstairs. On a 1.5Ah battery it sucks for a bit less than ten minutes, which turns out to be plenty of time.
So: who made that?
Don't care. I'm waking up warm at night.
Also: me, obviously. ChatGPT doesn't have a fucking furnace.
(let's assume that)My 3dprinter is special. It has a bunch of values + an algorithm (i.e. a neural network) that takes input as tokens and outputs a printed object.
I would say „I made this gadget with my 3d printer, but the designer is someone else (I found the model online)“. The intent, the drive, the action comes from the human
"I made this proof myself, but the designer is someone else" is an extremely unconvincing claim to ownership.
Almost as if a proof isn’t the same as a 3d print. It’s just not a good analogy
> Do you attribute the build to the tool? The "system's contribution" is helped by many other things all the way down to chips, datacenters and power generation. If the authorship requires attributing to a tool, then it should happen all the way down.
When the tool is a 3D printer, or any CNC system really, you bet I attribute a build to it.
I could also attribute the operator; there is no contradiction, it's a free choice, just like saying "I am in Berlin" does not contradict "I am in Germany".
> AI has no self-awareness
What is your mechanistic model of self awareness that yields this conclusion?
> It's a tool
Does your model suggest that tools can't have self awareness?
Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to, and influences future choices. This implies statefulness, which models are intentionally not at inference time (*).
(*) Even if we hack around this and just do the usual trick of simply laundering statefulness to a higher level, in this case the context window being fed in, I fail to identify (**) a representation of its own state in these bodies of text that it'd be meticulously maintaining. I further fail to identify how it could be hidden or maintained, considering I control like half of it. The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
(**) I did wish for something hidden though, so maybe it's just concealed? The same way people can encode a lot more of their emotional and mental state than normal into text if they read and write a lot of it, I'm aware of research that suggested the same for LLMs, albeit I cannot cite it. Maybe those phrasing signatures are just alien to me and will never pop out. Either way, I'd expect researchers to stumble upon this during interpretability studies, and either they haven't, they have but it wasn't popsci adopted, or they're keeping awfully tight lipped about it. If you know of anything like this, your turn now, would be happy to learn.
I do wonder how reasonable it is to expect e.g. a single maintained identity though. Maybe it isn't?
(*) Another way to hack around this of course is to just precompute some internal "self-awareness states" and hop around between them. Probably the closest to what the models are actually "doing".
Before reading, know that I am uncertain in either direction.
> a hidden representation of self that is continually tended to
This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it.
> I further fail to identify how it could be hidden or maintained, considering I control like half of it.
Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output.
> The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one".
> You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the end, though the person in question didn't ever get diagnosed with that.
> If you know of anything like this, your turn now, would be happy to learn.
Do you mean like these, or something else?
• https://researchportal.hkust.edu.hk/en/publications/decoding...
• https://aclanthology.org/2026.eacl-long.165/
• https://transformer-circuits.pub/2026/emotions/index.html
> This sounds like a personality?
Not quite what I meant, but it's also not entirely unrelated I guess? Personality to me is like a natural bias. It does also shift over time, and is also an internal bit of state. I guess in some respects it can also be self-referential, like personal convictions.
> Perhaps they were losing their self-awareness at the time?
I do think it is entirely possible for people's self-awareness to shift, yes. Or more precisely, I do model things that way.
> Do you mean like these, or something else?
They're adjacent, but I more meant something like these:
https://arxiv.org/abs/2410.03768
https://arxiv.org/abs/2310.18512
https://arxiv.org/abs/2605.26537
So basically, steganography. The difference is that these papers investigate from the perspective of separate LLM instances covertly exchanging information between each other. This is in contrast with the scenario I'm laying out, where an LLM's past state is exchanging information with its future state, continuously representing and modulating a concealed internal state of some sort. And then that state just so happening to be some sort of self-referential meta state.
And the best inkling I have towards this is basically: https://www.youtube.com/shorts/WP5_XJY_P0Q
But then I don't think there's enough covert channel bandwidth in the agent replies for anything interesting like this.
> Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to
I don't see why an LLM could not have a sense of identity or personality while it's evaluating a specific prompt, or even change self awareness while evaluating a prompt since many outputs model a back and forth conversation. My point is that without a mechanistic model of what "self awareness" means, we have no way of truly evaluating such questions, we're just hand waving vague intuitions about what it could mean.
Sure, but then such a model is not going to make itself. People pitting their vague intuitions is how such models eventually form. I'd also push back regarding that my comment would have been handwavey or without mechanistic elements, even if it was on the whole informal.
This is kind of also the reason e.g. the HN site guidelines are worded the way they are. Regrettably, forums naturally yield themselves to tit for tat type exchanges, but there's really no reason one could not bounce such vague intuitions off of another. I do not have to be right or wrong, and you don't either. Admittedly difficult when its some intensely contentious topic.
If a mechanistic model existed, there would also be no reason to talk about this in the first place. There'd be nothing to discuss, you'd be simply told how a given model characterizes from this perspective on the model cards.
Even mechanistic models generate interesting discussion. How many years have we discussed Turing machines and the lambda calculus? Almost a century of great work came out of those.
The reason I insist on mechanistic models is because the original post was making a definitive knowledge claim, and in my experience, the knowledge claim is unwarranted.
I honestly don't think a language model is enough for self-awareness, regardless of the exact model of awareness.
A language model (or an image model or whatever) cannot even be sentient, and I think sentience is a prerequisite for awareness.
Even if we express a lot of our subjective experience with words, the language is just a symbolic representation of those experiences. The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution.
You can't have an understanding of what hunger or physical pain feel like if you have no need for food or a sensory capacity for feeling pain. You can't understand what loneliness or pride at an achievement mean if you don't have a neural network wired to value social connection or status. We value connection because we're social animals that have needed each other for survival.
Even the more abstract of our subjective experiences are in some way rooted in our physical evolution.
I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
AI awareness might actually be more believable if that awareness manifested itself in an entirely different way than in humans. But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience, I think we're seeing something that isn't actually there.
> The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution.
There is no objective evidence of qualia. All evidence of qualia are vocal or other expressions of belief in qualia. Perceptions clearly exist and are observable, subjective experience and qualia, not so much.
> I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
If your objection is to models based on "symbolic level of language" which you think lack semantic understanding of, say, trees, you should ask yourself how our brain, based on physics which also lacks any semantic category for trees, can somehow develop a semantic understanding of trees. All of these appeals to differences with the brain never seem to acknowledge that fundamentally, the brain has the same explanatory gap with physics.
> But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience
This assumes a lot. It seems very possible to me that intelligence inherently develops a map of natural categories (natural kinds), and language naturally develops around such categorical understanding. Semantics are then fundamentally the network of associations between categories, eg. there is no fundamental difference between symbols and semantics, and the latter cam be inferred from the former, and that's exactly what LLMs do, and why the semantic maps between different languages are so similar and how they can translate between languages.
So… your model is 100% vibes based. Got it.
I wasn't trying to give a model. The point was that I don't think it's necessary to give one.
You didn't address any of what I wrote, let alone provide any counterarguments. Which part of what I wrote do you think was wrong?
Making definitive claims about whether LLMs do or do not have specific properties absolutely does require precise definitions of those properties that can be used to evaluate those questions. Merely hand waving that LLMs didn't undergo the same evolutionary process is not a definitive argument.
For example, the Turing machines and the lambda calculus don't look anything alike, but they are fundamentally interconvertible, and so in a real sense they are fundamentally equivalent. Without a model, all of your arguments are completely unconvincing for exactly the same reasons, eg. that there may exist many paths to fundamentally equivalent ends.
I just don't think linguistic (or other symbolic) representations alone can contain the information, in any sense of the word, of what e.g. human subjective experiences actually are like. The concepts we express with language get their meaning from our physical reality, even if quite indirectly in case of some abstract concepts.
Hunger as a concept doesn't mean anything without the physical need. Politeness or bluntness, even in writing, don't mean anything without social dynamics. And we have social dynamics (and neural structures that directly process social cues and associated feelings) because we've evolved into social animals for whose survival that was important.
I see no reason to believe that a model trained only with symbolic representations, with no connection to the physical world phenomena that those symbols represent, could contain the subjective experience itself.
Neural network models may be able to derive novel (or at least novel-looking) output rather than just an obvious rehash of their input, but I don't think any set of bytes can fundamentally contain information that was never entered into it. (Even if e.g. a model produces previously unknown mathematical results, those results can in principle be derived from the information that they were trained with.)
I'm not saying that artificial neural networks couldn't, in principle, be aware. ANNs and biological neural nets may be equivalent in the sense that any information and processing structures represented by a biological one could in principle be represented by an artificial one. If that's the case, and awareness is purely a product of our neural systems as materialism would imply, it should be possible for an ANN to be aware, too.
But when the model has been trained with only language, and IMO the subjective experience can't be derived from the symbolic representation alone, I can't see how the model could include the actual subjective human experience.
An AI model could of course have an awareness and subjective experiences that are totally different than our human experience. But then the fact that it happens to produce output resembling what humans find meaningful shouldn't be considered indicative of such awareness.
This is of course more of a philosophical argument than a technical one, and I'm happy to hear counterarguments, but not on the level of off-hand dismissal.
> I can't see how the model could include the actual subjective human experience.
People who say LLMs have subjective experience aren't saying they have human-type subjective experience. Nobody who sees an LLM express hunger when role playing as a hungry person thinks that the LLM is actually hungry.
I too can role play as a hungry person despite not being hungry, so there is no reason in either case to conclude that the words produced reflect genuine internal subjective states. The point is that such internal states may still exist.
How do we know that these solutions don't exist in the training data? It is open secret that they have used pirated materials for training. Perhaps it plagiarized solutions from works of some obscure Belgian mathematician from the sixties, who did not get mainstream acceptance. I wouldn't be surprised if they also got access to mathematics done in the "defense contractor" setting from various three letter agencies.
Without a searchable index of training data, it is hard to put faith into these claims.
upvoted for fair point .. its possible an LLM AI could be put to work to search widely for attribution / similar results.
eg. "we spent another 2k on searching for pre-existing proof but found only the weaker result xyz by abc in 1972" would be in the spirit of academics quoting prior work.
This wouldn't be a problem so long as they properly attribute.