I'm not an expert, but nothing, I think, is stopping it in principle. I suspect there's still a lot of practical problems relating to precision, usable skills, affordable components and ability to scale and any number of devilish engineering and physics problems. Just a whole domain of problems related to physical interaction over and above issues of interfacing with AI.
But in principle I think physical automation is the next big frontier and its probably coming within our lifetimes.
Nothing, but to get something that garners a reaction more than just "that's really cool", and instead, "that's really cool, and also terrifying", you need to up your inference speed.
This will come in the form of models that are baked into silicon. AMD recently acquired Taalas, which is a company that specializes in this stuff. They've shown some really promising proof of concepts that result in a nearly 50x increase in token output speed.
Once we see it come to market, things will begin getting really crazy. Not sure how I'm feeling about that.
I am not an expert on this but from what i have seen, putting an llm into a physical robot requires a combination of good vision capability, good physics deduction and stuff. it's not just reasoning, i think the model can reason fine about how it needs to stuff in the real world, but imho, it's more like llms don't have that level of spatial cognition yet that say humans intuitively have, speed is also definitely a factor though.
The reasoning ability is not what is missing. Claude could very easily deduce that it needs to walk to the other room, pick and object up and bring it back based on the prompt.
The difficulty is then mapping that goal of "move to other room, grab, move back" to actual outputs to some motor system. And "pick up object" is much too complex to be compared to a simple tool call in some AI agent.
It is one thing for an LLM to understand when it needs to run a bash script to collect some system info. It is another thing entirely to expect it to be
able to take an image, process it, understand depth/distance, map that delta to some co-ordinate system, then call the exact correct motor functions with the right parameters to cover that distance, ( not accounting for positioning hand to correctly grip and pickup the object of arbitrary shape) grab the item and return home.
Doing this using traditional preset instructions and training data is not impossible, but if you mean a truly dynamic AI driven robot, there is just so much variance we arent at a point where it can be consistently producing realistic/desired results (commercially of course.) though I don't think we are very far from it.
TLDR, agentic AI for programming today is still leaps and bounds behind a consistent realistic AI body. Many many many moving pieces where the hallucinations and variance in modern LLMs largely produces inconsistent results. I am not an expert, this is just from anecdotal experience working with very simple robotics and LLMs.
I'm not an expert, but nothing, I think, is stopping it in principle. I suspect there's still a lot of practical problems relating to precision, usable skills, affordable components and ability to scale and any number of devilish engineering and physics problems. Just a whole domain of problems related to physical interaction over and above issues of interfacing with AI.
But in principle I think physical automation is the next big frontier and its probably coming within our lifetimes.
Nothing, but to get something that garners a reaction more than just "that's really cool", and instead, "that's really cool, and also terrifying", you need to up your inference speed.
This will come in the form of models that are baked into silicon. AMD recently acquired Taalas, which is a company that specializes in this stuff. They've shown some really promising proof of concepts that result in a nearly 50x increase in token output speed.
Once we see it come to market, things will begin getting really crazy. Not sure how I'm feeling about that.
I am not an expert on this but from what i have seen, putting an llm into a physical robot requires a combination of good vision capability, good physics deduction and stuff. it's not just reasoning, i think the model can reason fine about how it needs to stuff in the real world, but imho, it's more like llms don't have that level of spatial cognition yet that say humans intuitively have, speed is also definitely a factor though.
The reasoning ability is not what is missing. Claude could very easily deduce that it needs to walk to the other room, pick and object up and bring it back based on the prompt. The difficulty is then mapping that goal of "move to other room, grab, move back" to actual outputs to some motor system. And "pick up object" is much too complex to be compared to a simple tool call in some AI agent.
It is one thing for an LLM to understand when it needs to run a bash script to collect some system info. It is another thing entirely to expect it to be able to take an image, process it, understand depth/distance, map that delta to some co-ordinate system, then call the exact correct motor functions with the right parameters to cover that distance, ( not accounting for positioning hand to correctly grip and pickup the object of arbitrary shape) grab the item and return home.
Doing this using traditional preset instructions and training data is not impossible, but if you mean a truly dynamic AI driven robot, there is just so much variance we arent at a point where it can be consistently producing realistic/desired results (commercially of course.) though I don't think we are very far from it.
TLDR, agentic AI for programming today is still leaps and bounds behind a consistent realistic AI body. Many many many moving pieces where the hallucinations and variance in modern LLMs largely produces inconsistent results. I am not an expert, this is just from anecdotal experience working with very simple robotics and LLMs.
People are doing this.
I've seen many videos of LLMs driving arms and other robots.
DrivingBench even lets them drive a Tesla