AI in robots: perception, learned control and foundation models

Robots have used algorithms for decades, but machine learning is changing what they can do outside tightly controlled factories. Here is where AI appears in a modern robot.

Perception

Neural networks turn camera and depth images into useful information: which objects are there, where people are, what the floor looks like. This is the most mature use of AI in robots and is found in picking systems, service robots and inspection robots.

Learned locomotion

Many recent quadrupeds and humanoids walk with controllers trained by reinforcement learning in simulation: the robot practises millions of times in a virtual world with varied terrain and disturbances, and the resulting controller is transferred to the real robot. This has made legged robots much more robust on stairs, slopes and uneven ground.

Learning from demonstrations

With imitation learning, a person demonstrates a task, for example by teleoperating the robot or moving it by hand, and a model learns to reproduce it. This lowers the effort to teach new manipulation skills, especially for tasks that are hard to program by hand, such as folding cloth.

Foundation models for robots

The newest approach trains large models on huge amounts of robot and internet data so that one model can follow instructions in natural language and perform many tasks. Several humanoid makers build their products around such models. They are promising but still an active research area: reliability, speed and safety in real environments vary a lot between demos and daily use.

What to check when a robot is "AI-powered"

• What exactly the AI does: perception, navigation, locomotion or task learning.
• Whether it runs on the robot or needs a cloud connection.
• Which computer is on board (see the compute platform in the specifications).
• Whether you can train or change the models yourself.

Explore humanoids and research robots, and read the known issues owners report in real use.