Creating realistic streets with GenAI

Automatic removal of vehicles from photorealistic 3D street scenes – directly in the 3D Gaussian Splatting (3DGS) representation, without destroying the reconstruction through 2D inpainting.

Background

Virtual driving school environments benefit greatly from realistic, photorealistic 3D reconstructions. However, video recordings of real streets almost always contain "transient" objects such as parked vehicles that interfere with training scenes. An obvious approach is to remove vehicles from the video (inpainting) and then calculate the 3D reconstruction – but in practice, this often leads to massive errors in the Structure-from-Motion pipeline, because synthetic inpainting textures are not consistent between frames.

Problem Statement / Goal

The goal is a robust, fully automatic method to remove vehicle Gaussians from an existing 3DGS scene – selectively and geometry-preserving: buildings, road surfaces, sidewalks, and markings should remain intact.

Approach

The approach combines (1) vehicle detection in 2D (YOLOv8) with (2) multi-view projection of detections into 3D space via COLMAP camera parameters and (3) a three-stage filtering strategy to address 2D→3D depth ambiguity:

• Layer 1 – Bounding Box Projection: coarse candidates within the 2D box.
• Layer 2 – Depth Percentile Filter: only the nearest Gaussians (e.g., 30%) within the box → protects background facades.
• Layer 3 – Height Threshold: removes only Gaussians above ground level (e.g., 30 cm) → protects road and sidewalk.
• Multi-View Consensus: a Gaussian is only removed if confirmed in ≥2 views → reduces false positives.

Results

Experiments showed: video inpainting before COLMAP can drastically degrade reconstruction. 3D-level removal after training, on the other hand, preserves the scene and selectively removes vehicle components.

Key Takeaway
"For reconstruction pipelines, it is more reliable to edit objects in 3D rather than in 2D – this preserves all original information for COLMAP/3DGS."

Benefits

The approach enables capturing real street scenes even with normal traffic/parking activity and then automatically cleaning them up – without manual masking and without hours of re-optimization. The cleaning runs as post-processing on the explicit 3DGS format (PLY) and is therefore very well integrable into a content pipeline for VR/simulator scenes.

Key terms

• 3DGS3D Gaussian Splatting
• Object RemovalVehicle removal / Scene editing
• Multi-View2D→3D projection + consensus voting
• COLMAPCamera parameters from SfM
• YOLOv8Vehicle Detection

Industry Partner

SimuDrive
(Project context: virtual driving school / driver training in VR)

Team