Subtitles for Real Life? Augmented Reality Subtitles using Audio-Visual Speech Recognition and Real-Time Translation
Can AR glasses display real-time subtitles of spoken German in noisy environments? This project investigates how combining lip-reading with audio recognition improves speech-to-text accuracy in restaurants, parks, and busy streets and explores the feasibility of a full subtitle pipeline on Microsoft HoloLens 2.
Concept illustration of real-time AR in a restaurant
Initial situation
Language barriers remain a significant challenge in multilingual settings such as international meetings, travel or everyday conversations. While automatic speech recognition (ASR) has made great progress, most systems rely solely on audio and struggle in noisy real-world environments like restaurants or busy streets. Audio-Visual Speech Recognition (AVSR) offers a promising approach by incorporating visual cues such as lip movements to improve robustness under noise. At the same time, Augmented Reality (AR) devices like the Microsoft HoloLens 2 enable new ways of presenting translated speech as real-time subtitles directly in the users field of view. However, combining AVSR with machine translation and AR visualization into a single working pipeline has not yet been thoroughly explored, especially for German in realistic acoustic conditions.
Problem statement / Project goal
The goal of this project is to evaluate whether audio-visual speech recognition can be used to generate real-time AR subtitles for spoken German, even in challenging acoustic environments. Specifically, the project addresses three research questions: (1) How do real-world noise environments like restaurant, park and street affect the accuracy of AVSR compared to audio-only recognition across different noise levels? (2) Can an adaptive strategy that selectively activates visual processing based on noise conditions improve efficiency without sacrificing accuracy? (3) What are the practical constraints for deploying a complete speech recognition, translation and AR visualization pipeline on a HoloLens 2?
Noise robustness evaluation
We evaluated how restaurant, park, and street noise across five intensity levels (−10 to +10 dB SNR) affect recognition accuracy. Performance degraded consistently with increasing noise, with restaurant environments being the most challenging. Audio-visual input improved robustness in difficult conditions, but gains were not consistent across all environments and were affected by occasional high-error outliers.
Adaptive audio-visual fusion
We investigated an adaptive strategy that activates visual processing only when audio confidence is low. This approach reduces unnecessary computation while retaining most of the robustness benefits of audio-visual models, enabling a better trade-off between accuracy and efficiency in dynamic noise scenarios.
AR pipeline feasibility
The envisioned end-to-end pipeline would capture speech, perform recognition, translate the output from German to English and vice versa, and display subtitles on the HoloLens 2. We assessed the feasibility and latency constraints of deploying such a system in real-world scenarios.
Solution developed and its benefits
We evaluated the core components needed for AR-based real-time subtitles: audio-visual speech recognition under realistic noise, an adaptive fusion strategy and the feasibility of deploying such a pipeline on the HoloLens 2. The AVSR models were benchmarked using the MuAViC German dataset augmented with real-world noise samples from FreeSound.org, covering three representative environments at multiple noise levels.
- Systematic comparison of Whisper (audio-only) and mWhisper-Flamingo (audio-visual) under realistic noise conditions
- Environment-specific noise augmentation pipeline for restaurant, park and street scenarios
- Adaptive routing strategy that selectively engages visual features based on audio quality
- Computational cost analysis showing the trade-offs of always-on vs. adaptive audio-visual processing
- Dynamic noise profile experiments simulating real-world fluctuations within single utterances
- Feasibility assessment of deploying AVSR with translation and AR subtitle display on HoloLens 2
The results show that audio-visual recognition provides notable accuracy improvements under noisy conditions, with the largest gains at low SNR levels. The adaptive routing strategy offers a practical middle ground by reducing computational cost while preserving much of the accuracy benefit. The live pipeline on HoloLens 2 performs reliably under moderate noise conditions, but the model tends to hallucinate under strong background noise, producing unreliable subtitle output.
These findings demonstrate that AVSR-powered AR subtitles are feasible for real-world use, while highlighting that robustness under extreme noise remains a key challenge for future work.
Key terms
- AVSRAudio-Visual Speech Recognition — combining audio and lip-movement analysis for robust speech-to-text
- WhisperOpenAI's multilingual automatic speech recognition model, used as the audio-only baseline
- mWhisper-FlamingoA multilingual audio-visual extension of Whisper that integrates visual features via AV-HuBERT
- MuAViCMultilingual Audio-Visual Corpus — the benchmark dataset used for German speech recognition experiments
- SNRSignal-to-Noise Ratio — a measure of how much background noise is present relative to the speech signal
- WERWord Error Rate — the primary metric for evaluating speech recognition accuracy
- Adaptive AVA routing strategy that activates visual processing only when audio quality is insufficient
- HoloLens 2Microsoft's mixed reality headset, targeted as the deployment platform for AR subtitles
- Augmented RealityTechnology that overlays digital content onto the real world, used here for subtitle display
Team
Manjavy Kirupa
Shwetav Behera
Advisor
Marianne Santoholma
Vagia Tsiminaki
Arzu Çöltekin