The future will be subtitled

A few years ago, Microsoft HoloLens and Magic Leap promised a future of groundbreaking visual experiences that blend digital and real worlds. While the headsets were bulky, expensive, and proprietary — and, in fact, both products are dying slow deaths — the demos got the public used to the idea that the future of augmented reality would be wild and filled with 3D visual content. 

HoloLens projected interactive 3D holograms into users’ environments, allowing them to manipulate these holograms with natural hand gestures, eye tracking, and voice commands. Public demos showed fighting virtual robots in the living room and giant anatomical models for education.

Magic Leap showed off hyper-realistic digital humans like its AI assistant, Mica, who could recognize a user’s mood and interact as though present in the room. The company promised architectural walkthroughs and collaborative design sessions, where multiple users could manipulate big 3D models together.

Glorious stuff. But the market rejected these extremely visual appliances. 

My prediction is that not only will augmented reality (and heads-up displays) go mainstream fast, the most common use case for the visual element will be: subtitles.

The incredible power of captioning

Polls from CBS News and Preply found that more than half of home movie and TV watchers watch with subtitles on. A 2024 survey found that 70% of Gen Z adults (ages 18–25) and 53% of Millennials (ages 25–41) watch most of their online video content, including on YouTube, with captions or subtitles enabled.

The preference for captions or subtitles in scenarios where the user largely controls the ambient sound and also the volume of the media suggests that when captions are available in smart glasses or AI glasses, much of the public will probably choose them to be on most of the time, especially in situations where context is less clear. 

Connected AI glasses that can display visual content to the wearer and which can pass for ordinary, everyday glasses are ideal for providing subtitles to the world. Microphones in the glasses can listen, cameras can watch, and AI can interpret and provide its interpretation in written language visible to the wearer and to nobody else. It’s very powerful stuff.

Here are the many uses for captioning in smart glasses:

1. Hearing aids

While Apple is looking to transform AirPods into hearing aids for the hearing impaired, other companies are doing something better: providing captions for the hearing impaired.

East Coast companies Vuzix and Xander are now shipping captioning glasses for the hearing impaired. Based on Vuzix’s M400 smart glasses and Xander’s software, the product works by picking up speech through built-in microphones and then instantly transcribing and projecting the words onto the lenses. The system runs entirely on the glasses, so it doesn’t need a phone or internet connection. The glasses are available for audiologists and clinics to purchase and use with patients.

2. Language translation

Google demonstrated the power of captioning for language translation three years ago with an experimental prototype that has since been discontinued. Google’s highly produced video showed a perfect example of how captioning would enable conversations between two people who don’t speak the same language.

Much more recently, Meta rolled out its previously experimental Live Translation feature to all Ray-Ban Meta users. I use Ray-Ban Meta glasses and live translation in Mexico, Spain, Italy, and France (I’m currently in Italy), and it falls short precisely because it does not have captioning. The feature speaks the translation in your ear and also types it out in the app. Another bonus is that it translates my English into the foreign language I’m interacting with via the app. 

This would be amazing if I could see all the translation happening in my glasses. 

By the way, I have also been using Meta’s sister feature, Live AI. With Live AI turned on, I can ask the Meta assistant what foreign language signs mean in English, and it tells me. Imagine having all foreign language signage displayed in your own language at all times while abroad. What’s amazing about this is that, unlike Live Translate, I don’t have to tell the glasses in advance what languages I’m working with. It will translate Japanese signs as quickly as it does French or Portuguese ones. 

3. Speaker notes

Shahram Izadi unveiled Google’s new Android XR platform at a recent TedTalk; while he focused on many groundbreaking applications that blend AR with AI, he also pointed out that he could see his speaker notes in his prototype glasses. 

In addition to notes during speeches and presentations, captioning will give media types and politicians teleprompter capabilities in their glasses during speeches, TV hits, and podcast appearances. 

4. Tours and tourism

It’s common for museums to rent headsets to listen to while browsing collections and for tourists to use various media to get contextual information while wandering around on vacation. AI and AR are ideal for this kind of application, where tourists can learn all about any neighborhood, cultural artifact, or museum display with silent captions in glasses. 

5. Content consumption

And finally, we come full circle. With captioning in AR glasses, two people could watch a TV show, movie or YouTube video, and one could get captions while the other person could choose to not see them. 

That also applies to lyrics displayed during concerts, language translation during Italian operas, song identification while at bars and restaurants, and other sorts of context spelled out in noisy settings.

The whole point of augmented reality is to provide useful information about the world which the world itself isn’t clearly providing. And while we’ve been dazzled by incredible visuals, the truth is that the best way to augment reality will almost always be with captions and subtitles.