Sparse Input Gaussian Splatting for 3D Facial Reconstruction and Augmentation of Stable Diffusion based Person Generation
This project uses 3D face reconstruction via Gaussian Splatting to generate additional views from just five photos, improving input diversity for Stable Diffusion.
We use Input Images to reconstruct a 3D Object of the face and then generate Images with Stable Diffusion from new synthetic views.
Initial situation
At the beginning of 2024, an AI selfie box was installed at the Enter Technikwelt museum in Solothurn, developed in cooperation between the Institute for Data Science and the Institute for Interactive Technologies at the FHNW. In this box, five photos of a person are taken from different angles and transferred to a Stable Diffusion DreamBooth model together with a prompt. In this way, the person shown can be presented in a desired environment or in a defined context like as a jedi or a hero.
Problem statement / Project goal
The central problem addressed in this project is the limited number and variety of angles of the input images, which could restrict the quality of the portraits generated by the model. The aim is therefore to augment the existing images in a targeted manner in order to improve the capture and reproduction of the person's appearance in the diffusion model.
3D Gaussian Splatting Pipeline
To achieve the best possible 3D reconstructions of facial input images, we constructed a Gaussian Splatting pipeline featuring multiple extensions, like calculating the camera positions with Structure-from-motion or Depth Map generation via pretrained Neural Networks.
3D Gaussian Splatting Reconstructed Scene
Example of a 3D Gaussian Splatting Reconstruction of one of the Project Members. The whole Scene is build by many optimized Multi-Dimensional Gaussian Functions which together blend in to one 3D Object/Scene.
3D Gaussian Splatting Reconstructed Scene
Example of a 3D Gaussian Splatting Reconstruction of one of the Project Members. The whole Scene is build by many optimized Multi-Dimensional Gaussian Functions which together blend in to one 3D Object/Scene.
Solution developed and its benefits
To enhance person-specific image generation with limited input, we developed a custom pipeline combining 3D Gaussian Splatting with Stable Diffusion fine-tuning. Our approach reconstructs a face in 3D from just five images, generates novel views through interpolation, and feeds these into a DreamBooth model for better portrait quality. Key improvements include:
- Depth-based regularization and human segmentation, which significantly reduced floating artifacts and background noise in reconstructions.
- A view interpolation strategy, allowing us to add up to 16 synthetic perspectives, boosting data diversity without requiring extra hardware.
- A tailored evaluation process, combining subjective self-assessments and objective metrics (e.g. PSNR, cosine similarity of Feature-Embeddings with a Face Recognition Neural Network), to measure the recognizability and fidelity of the generated outputs.
- Users rated the 21-view augmented models highest, with recognizability improving by ~25–30% over the base model.
- The additional reconstruction step added only a few minutes of processing time—well within the acceptable window for real-world museum deployment.
- Our findings support Gaussian Splatting as a practical augmentation tool for person-specific generation, particularly when hardware constraints limit direct data capture.
Benefits:
Segmentation Map of one of the Project Members
We used a pretrained Neural Network to generate a Human Segmentation Map which tells us which pixel belongs to a Human. This enabled us to simply remove the background for a few specific tasks, giving the model the opportunity to focus on the person.
Depth Map of one of the Project Members
We used a pretrained Neural Network to generate a Depth Map which tells us how far every Pixel is from the Camera. This enabled us to achieve the more accurate 3D reconstructions, with no stray artifacts floating in the air where they don’t belong. Bluer pixels are closer to the camera, while redder pixels are farther away.
Example inputs for 3D Gaussian Splatting reconstruction
Example inputs for a part of our augmented 3D reconstruction pipeline. The val image was used to validate our model from a view which the model never saw. The three Images in the bottom are some output-views of the 3D Gaussian Splatting reconstruction, which we then used to fine tune the Stable Diffusion Model to generate new Images of the person.
Stable Diffusion generated Image of a Project Member with Gaussian Splatting Augmentation
An example of one of the final generated images.
Stable Diffusion generated Image of a Project Member with Gaussian Splatting Augmentation
An example of one of the final generated images.
Stable Diffusion generated Image of a Project Supervisor with Gaussian Splatting Augmentation
An example of one of the final generated images.
Key terms
- Gaussian Splatting3D Reconstruction Model which builds a scene from many small Gaussian Functions.
- 3D Face ReconstructionReconstructing a Human Face as a 3D Object
- Stable DiffusionAI Model which generates images with Text Prompts
- Depth MapMap which tells us how far every Pixel is from the Camera. A Neural Network generates this Map by predicting every Pixel of the input Image.
- Segmentation MapMap which tells us which pixel belongs to a Class (in this case a Human). A Neural Network generates this Map by predicting every Pixel of the input Image.
- DreamBooth Fine-TuningMethod to Fine Tune a Diffusion Modell on new classes or conditions
- Synthetic View GenerationGenerating Views from a 3D Scene which are not in the Trainset
- Data AugmentationChanging Input Data to get more different Data
- Structure-from-motionAlgorithm which computes Camera positions out of many overlapping input images with 3D Triangulation
Team
Jan Zumwald
Can-Elian Barth
Advisor
Michael Graber
Moritz Kirschmann