Virtual Driving School
Removing the ego vehicle from 360° street captures and restoring the road surface beneath it — a two-phase pipeline of temporal recovery and context-aware inpainting for 3D Gaussian Splatting scenes.
Initial situation
Virtual-reality driving simulators such as SimuDrive depend on street scenes that look and behave like real roads: correct geometry, readable signage, and — critically for a driving task — an authentic road surface with its real lane markings, crossings, and wear. Reconstructing such scenes by hand is prohibitively expensive, which motivates capturing real streets and reconstructing them automatically. 3D Gaussian Splatting (3DGS) has made photorealistic, real-time novel-view synthesis practical, and a 360° roof-mounted camera is an attractive capture device: a single forward drive records the full surroundings of the vehicle in one pass.
Problem statement / Project goal
A roof-mounted 360° capture has one structural defect for this purpose: the ego vehicle is present in every frame. In the reconstructed scene it appears as a smeared, incoherent blob directly beneath the camera trajectory, and the road surface it occludes — the nadir region — is never directly observed. For a driving simulator the result is unusable: the driver would look down onto a distorted mass of the capture vehicle instead of the road they are driving on.
The task of this project is therefore ego-vehicle removal and nadir inpainting for 360°→3DGS street scenes: remove the ego vehicle and fill the road beneath it with an authentic surface — real road markings where they can be recovered, plausible generated asphalt only where they cannot.
Solution developed and its benefits
The developed pipeline addresses the problem in two phases. First, a temporal-aggregation step recovers real road surface from the captured video by exploiting vehicle motion: a ground point occluded by the ego at one moment is often visible in the non-ego portion of a neighbouring frame. Second, a context-aware video inpainter (ProPainter) fills the residual pixels that vehicle motion never exposes, using surrounding road as context and producing frame-coherent output — not per-frame hallucinations that later confuse the 3D reconstructor.
- Recovers roughly 30.8 % of the road-plane cells under the trajectory from real video, without any generated content
- Shows that recovery is governed by local frame density, giving a concrete, actionable capture-protocol recommendation for SimuDrive
- Establishes on a pixel-accurate synthetic-hole benchmark that context-aware video inpainting preserves real road markings that per-frame inpainting erases
- Delivers a trained 3D Gaussian Splatting reconstruction with a filled, marking-bearing nadir surface as the production output
The result is a 3DGS scene that can be navigated in a driving-simulator context without the ego-vehicle artifact and with an authentic road surface directly beneath the driver's viewpoint.
Testimonial
«Mit der IP6 Arbeit wurde ein wichtiges Puzzleteil zum Gesamtprojekt mit eingebracht.»
“The IP6 project contributed an important piece of the puzzle to the overall project.”
Ivan Pugneth, Founding member and CEO, SimuDrive GmbH
Key terms
- 3D Gaussian Splattingexplicit primitive-based radiance-field reconstruction for real-time novel-view synthesis
- Ego-vehicle removaleliminating the capture vehicle from a roof-mounted 360° recording
- 360° street captureequirectangular video from a roof-mounted omnidirectional camera
- Temporal aggregationrecovering occluded ground by using neighbouring frames where the same ground was exposed by vehicle motion
- Video inpaintingcontext-aware, temporally-coherent generative fill (ProPainter, LaMa)
- VR driving simulatorthe SimuDrive downstream target application
Customer
SimuDrive GmbH
Ivan Pugneth, Founding member and CEO
Team
Nicolas Ostwald
Advisors
Prof. Dr. Arzu Coltekin
Marianne Santaholma
Oscar Alonso