If you hear a crashing sound and at the same time see a glass cup shattering on the floor, it will be very clear to you that the sound resulted from the breaking glass. Making these types of connections comes so naturally to us that we see them as obvious and take it for granted that we can so easily recognize them. But that is not the case for machines. Computer vision algorithms, for example, start with no knowledge of the world and have to learn everything from scratch — and there is an awful lot to learn.
So much to learn, in fact, that simply showing the algorithm lots of examples of real-world events is a losing game. It is completely impractical to provide enough examples to teach it to understand everything that it might potentially encounter. So rather than throwing more data at machine learning models, they have to be designed to better understand the world around them from the ground up.
An overview of CAV-MAE Sync (📷: E. Araujo et al.)
A team led by researchers at Goethe University of Frankfurt and MIT has just proposed a new approach called CAV-MAE Sync that seeks to solve the problem of associating sounds with the visual elements that caused them.
The new system is an evolution of a prior model called CAV-MAE, which was designed to learn from video and audio data without relying on human annotations. Unlike previous methods that treated entire audio segments and video clips as a single unit, CAV-MAE Sync breaks audio into smaller temporal chunks. This allows the model to align specific video frames with the exact audio events that occur simultaneously, producing much finer-grained understanding.
This mimics the way humans naturally connect what they see with what they hear. For example, when watching someone play a cello, we can instinctively identify that the movement of the bow across the strings is the source of the music. By training artificial intelligence systems to make similar connections, the research team is advancing machines toward more human-like perception.
Shorter audio segments contributed to the improved performance (📷: E. Araujo et al.)
CAV-MAE Sync leverages two learning objectives to balance the model’s learning process. One objective is contrastive learning, in which the model is trained to associate similar audiovisual pairs. The other is reconstruction, or teaching the model to recreate original audio or video data based on learned representations. Traditionally, these goals interfere with each other because they operate on the same underlying data. CAV-MAE Sync resolves this by introducing two new types of data tokens: global tokens for contrastive learning and register tokens for reconstruction. This separation gives the model more flexibility and leads to better performance.
The result is a system that not only better associates sound with visual events but also performs well on a variety of other tasks. In tests on widely-used datasets such as AudioSet, VGG Sound, and ADE20K Sound, CAV-MAE Sync achieved state-of-the-art results in video retrieval, classification, and localization — even outperforming more complex models that require more training data.
Looking ahead, the team aims to extend the system to incorporate text data, with the goal of developing an audiovisual large language model. By enabling machines to process audio, visual, and textual information together, the researchers hope to create AI that perceives and understands the world more like we do — not through more data, but through smarter design.