MIT and MIT-IBM Watson AI Lab help machines see better using spatial acoustics

(Image Courtesy: MIT News)

Researchers at MIT and MIT-IBM Watson AI Lab have created a technique that allows machines to better perceive their immediate environment.

This is done using a machine-learning model that accurately captures the way sound interacts with an environment. Using the captured acoustic information, the system can then remodel an accurate representation of the environment like how, "humans use sound when estimating the properties of their physical environment".

In a blog post, the researchers say that the system analyses how humans perceive sound and is able to calculate the sound's source, its distance or any obstacles present between the source and its intended path.

The researcher's pitched its potential applications in virtual and augmented reality, but it could also be used to help AI's understand the world around them.

"For instance, by modeling the acoustic properties of the sound in its environment, an underwater exploration robot could sense things that are farther away than it could with vision alone," says Yilun Du, a grad student in the Department of Electrical Engineering and Computer Science (EECS) and co-author of a paper describing the model.

“Most researchers have only focused on modeling vision so far. But as humans, we have multimodal perception. Not only is vision important, sound is also important. I think this work opens up an exciting research direction on better utilizing sound to model the world,” Du added.

Initially, the system was trained using techniques similar to visual learning models, but the researcher's found that vision models benefit from photometric consistency, i.e., an object will appear to look roughly the same from different angles.

This wasn't possible in sound models because acoustics vary greatly between distances, objects and obstacles. This was solved using the reciprocal nature of sound and the influence it has on geometry.

“If you imagine standing near a doorway, what most strongly affects what you hear is the presence of that doorway, not necessarily geometric features far away from you on the other side of the room," said Andrew Luo, grad student at Carnegie Mellon University (CMU) and co-author on the paper.

"We found this information enables better generalization than a simple fully connected network,” Luo added.