One of the first pieces of wisdom that parents impart upon their children is to look before they leap. The idea behind this saying is that a lot of trouble can be avoided by considering the consequences of your actions before carrying them out. And this age-old advice is not only applicable to us humans, but also to robots. Whether they are autonomous vehicles navigating through crowded streets or robotic arms performing delicate assembly work, considering consequences before acting is essential for safety, efficiency, and success.
However, giving robots the ability to predict the consequences of their actions is easier said than done. Us humans have an intuitive understanding of how the world works — what goes up must come down, objects in motion tend to stay in motion, a dropped glass will likely shatter on a hard floor, and so on. This intuitive understanding, often referred to as world knowledge, allows us to make predictions about the outcomes of our actions. Robots, on the other hand, lack this innate understanding and must be explicitly programmed or trained to predict consequences, which can be a complex and challenging task.
But now, with Meta’s recent release of V-JEPA 2, a new world model built for visual understanding and prediction in the physical world, we are getting closer to the goal of giving world knowledge to machines. V-JEPA 2 has been shown to have state-of-the-art performance levels in this area, which could serve to enhance the physical reasoning capabilities of future AI agents.
V-JEPA 2 builds upon its predecessor, the original V-JEPA model introduced last year, by offering improved abilities in both understanding and prediction. Trained on massive amounts of video data, V-JEPA 2 helps AI agents interpret how humans interact with objects, how objects behave on their own, and how different elements in a scene affect one another. This level of understanding is crucial for enabling AI systems to “think” before they act, much like humans do.
Robots using this model have successfully performed real-world tasks such as reaching for and picking up objects, as well as placing them in new locations, even when encountering unfamiliar environments. The model’s strength lies in its ability to generalize from training data to novel scenarios, a key requirement for real-world deployment.
To encourage further development in this field, Meta has also released three new video-based benchmarks designed to evaluate how well models can reason about the physical world. These benchmarks aim to measure an AI’s ability to learn from video data, simulate possible outcomes, and plan accordingly — all key measures of physical reasoning. With any luck, robots will soon find themselves more at home in our world as a result of these efforts.