Google's Project Astra provides a glimpse at the future of AI assistants

At Google I/O 2024, the company showed off a two-part demo of an early prototype of Project Astra showcasing how it could work.

Google DeepMind CEO Demis Hassabis on May 14 unveiled Project Astra, an ambitious vision of what he believes will be the future of artificial intelligence (AI) assistants.

"For a long time, we've wanted to build a universal AI agent that can be truly helpful in everyday life. Our work making this vision a reality goes back many years. It's why we made Gemini multimodal from the very beginning," Hassabis said in his keynote at the company's annual developer conference Google I/O 2024.

"At any given moment, we are all processing a stream of different sensory information around us, making sense of it and making decisions. To be truly useful, these AI agents have to similarly understand and respond to our complex and dynamic world just like we do," he said.

"It has to remember what it sees and hears, better understand the context we are in, and respond quickly in conversation making the pace and quality of interaction feel more natural" Hassabis said. "It also needs to be proactive, teachable and personal, so users can talk to it naturally and without lag or delay."

Hassabis said that while they have made "great strides" in developing AI systems that can understand multimodal information, getting response time down to something conversational has been a difficult engineering challenge.

Read: Google introduces text-to-video AI model Veo to take on OpenAI's Sora

Building on the Gemini model, Google DeepMind has now developed AI agents that can process information faster, combine the video and speech input into a timeline of events, and cache it for efficient recall. The company has also enhanced how they sound, with a wider range of intonations using its models, Hassabis said.

During the conference, the company showed off a two-part demo of an early prototype of Project Astra (advanced seeing and talking responsive agent) showcasing how it could work.

In the demo video, a tester was walking through the office with the prototype running on a Google Pixel phone, asking the assistant to "see" for an object which makes sound, to which it responded with a speaker nearby.

It also answered a question about what an annotated part of the speaker is called and then subsequently interpreted the functionality of a particular code snippet displayed on a computer screen, all in real-time.

The assistant then correctly answered a question about which London neighborhood the office was located in, based on the view from the window, and also told the tester where she had left her glasses.

In another take, the tester was wearing prototype eyewear resembling smart glasses with the assistant answering questions in a hands-free mode. The assistant answered questions about a server flowchart, a hand-drawn reference to the Schrödinger's cat paradox and lastly picked a band name for a duo comprising a golden retriever and a toy tiger.

Read: Google unveils Gemini 1.5 Flash, an AI model for fast, low-latency applications

"With technology like this, it's easy to envision a future where people could have an expert AI assistant by their side, through a phone or glasses," Hassabis said. The company however didn't talk anything about these prototype glasses itself.

"As part of Android AR/XR, we are definitely working across the ecosystem. I think what you're seeing here is Project Astra really comes to life in a form factor like glasses, and so I think, it's definitely a direction we are investing in. We have nothing to announce on glasses at this time" said Alphabet CEO Sundar Pichai during a press briefing.

Hassabis said some of Project Astra's capabilities are coming to Google products, like the Gemini app and web experience, later this year.

This includes Gemini Live, a feature that will allow people to have in-depth, back-and-forth voice conversations with the Gemini assistant on the mobile app. One can even interrupt Gemini when it's talking so that the assistant can adapt to the user's speech patterns.

Google plans to bring video understanding capabilities from Project Astra to the Gemini app later this year, said Sissie Hsiao, vice-president and general manager, Gemini Experiences and Google Assistant.

"When you go live, you will be able to open your camera so Gemini can see what you see and respond to your surroundings in real-time," she said.

Gemini Live will be available to subscribers of Gemini Advanced, the company's paid AI chatbot tier, in the coming months.

Event alert: Moneycontrol and CNBC TV18 are hosting the ultimate event on artificial intelligence, bringing together entrepreneurs, ecosystem enablers, policymakers, industry leaders, and innovators on May 17 in Gurugram. Click here to register and gain access to the AI Alliance Delhi-NCR Chapter.