Approximation of global illumination in AR/MR

Experience realistic real-time global illumination running fully on-device on the Meta Quest 3. This project showcases a modular three-pillar rendering architecture combining live passthrough cubemaps, room-mesh shadows, and adaptive light probes to ground virtual objects in physical space, eliminating the floating sticker effect while maintaining smooth 72–90 Hz performance on mobile XR hardware.

Initial situation

In traditional 2D flat-screen rendering for desktop PCs and consoles, real-time global illumination has largely been solved through mature methods such as shadow maps, Screen Space Global Illumination (SSGI), and real-time path tracing. In Mixed Reality, however, the virtual camera operates in an unconstrained real-world environment where scene geometry and spatial lighting are unknown prior to runtime. Furthermore, standalone headsets like the Meta Quest 3 operate under severe power and thermal constraints on mobile System-on-Chip (SoC) architectures. When photometric and spatial relationships are ignored, virtual objects suffer from the "floating sticker effect". Without accurate contact shadows and dynamic occlusion, synthetic entities appear detached from physical ground planes and display unnatural static shading.

Problem statement / Project goal

The primary goal of this thesis is to identify and evaluate rendering techniques that enable believable object lighting in MR while adhering strictly to the constraints of target mobile hardware. The system aims to ensure that virtual objects interact convincingly with their surrounding physical environment without compromising real-time performance (72–90 Hz refresh rates). To demonstrate these concepts in a practical context, an MR application for viewing digital artifacts was developed as a practical showcase. The project also examines the computational and sensor limitations of modern consumer MR headsets and evaluates the feasibility of leveraging on-device sensor data to reconstruct an interactive lighting model.

Solution developed and its benefits

We developed a modular, three-pillar rendering architecture that brings real-time approximated global illumination to standalone Mixed Reality headsets across continuous pipeline stages: ingesting streaming feeds from passthrough cameras, un-projecting 2D pixels into a 3D environment cubemap using inverse pinhole projection matrices, healing unpainted blind spots through a dual-pass progressive Gaussian blur, constructing Z-buffer depth occlusion barriers and shadow-matte receivers from spatial room meshes, distributing adaptive 3D light probe grids across the physical volume, compressing radiance into low-order Spherical Harmonics, and evaluating localized diffuse shading on virtual objects. Two capture modalities were compared: a direct lighting integration combining passthrough feeds with AR light estimation parameters, and a low-bounce indirect capture sampling adjacent mesh surfaces.

• a complete real-time global illumination framework tailored for mobile XR hardware, no external physical light probes required.
• fully on-device processing: all passthrough video streams and spatial depth data remain strictly within local headset memory
• virtual objects rendered with realistic shadows and ambient reflections, eliminating the "floating sticker effect" for believable user perception
• every cast shadow and depth barrier cites the reconstructed spatial room geometry, so visual interactions can be verified against physical surfaces
• compute-heavy shader tasks are time-sliced and decoupled across consecutive frames, keeping frame rates smooth and preventing thermal throttling
• a three-tier fallback hierarchy that smoothly replaces unmapped camera regions with low-frequency ambient lighting gradients

On the evaluation setup, the best configuration executed the paint compute kernel in roughly 0.10 ms and the blur pass in 0.02 ms at 1024×1024 resolution. Applied to the Meta Quest 3 standalone headset, the pipeline processed live passthrough feeds and room geometry while comfortably maintaining target refresh rates of 72–90 Hz, establishing a solid foundation for photorealistic mobile mixed reality, ensuring that visual coherence can be achieved without requiring desktop PC performance.

Key terms

• Topicglobal illumination, MR, Meta Quest 3, Unity
• TechnologiesC#, HLSL

Team