Combining 2D Flow Matching and Gaussian Splatting for 3D Virtual Try-on

The work presents a pipeline for multi-view virtual try-on that generates realistic views of a person wearing new clothing from a 360° video and reconstructs a 3D model. It combines flow matching and Gaussian splatting.

Initial situation

Online fashion retailers often face the problem that customers order clothes that look good in the photos but do not like them after trying them on and therefore return them. This causes considerable costs and additional logistical effort for the shops, while for customers it means wasted time and frustration. One possible solution is to give customers the opportunity to “try on” clothes virtually directly on their own digital body to reduce returns of the product. Models that address this problem already exist. However, previous research has mostly focused on frontal views, while other perspectives have been neglected. In addition, existing research often concentrates only on individual aspects of the problem without developing a complete solution that combines all components into a practical overall pipeline. This forms the main motivation for our project objectives.

Problem statement / Project goal

This bachelor thesis aims to develop an end-to-end pipeline for a multi-view virtual try-on application that enables users to upload their own video and select a desired item of clothing. The system is designed to process not just a single frontal image, but a 360° video in which the camera rotates around the person, using generative models to synthesize the person wearing the selected garment consistently across multiple frames. Based on these generated views, the pipeline should produce a realistic 3D reconstruction of the person with the virtual garment, which can then be displayed in a viewer for continuous visualization from any angle.

Solution developed and its benefits

Overall, our pipeline successfully combines generative modeling and geometric reconstruction for multi-view virtual try-on. We show that high-quality virtual try-on training data can be generated without real paired datasets, and that unintended artifacts can be significantly reduced through prompt design, optical flow alignment, and fine-tuning. While strong improvements in 3D consistency were achieved, perfect multi-view consistency remains an open challenge. The final system produces recognizable 3D reconstructions, with 36 frames providing the best trade-off between quality and stability.

1. 1. First, a 360° video of a standing person is recorded. From this video, a set of frames is extracted at uniform intervals. The frames are then resized to a resolution of 1248 x 704 pixels.
2. 2. These frames are passed to VGGT, which predicts the camera parameters and an initialized sparse point cloud.
3. 3.1 Qwen Image Edit 2511 receives the frames together with a selected clothing item and a text prompt describing the editing task. The model then generates the new clothing on each frame. Qwen was fine-tuned for this task using LoRA.
4. 3.2 After each generation step, the generated image is processed by an optical flow algorithm that aligns it with the original input image. The aligned image is then used as the geometric reference for the next generation step in 3.1.
5. 3.3 After all images have been generated, the background is removed using SAM3 segmentation.
6. 4. Sapiens Depth is then used to generate depth maps for each image. Using the newly generated images of the person wearing the new clothing, a 3D reconstruction is performed with Gaussian Splatting. During training, the Sapiens depth maps are used to apply depth regularization. The final output of the pipeline is a 3D reconstruction of the person wearing the new clothing item.

Key terms

• Flow MatchingA training approach that is used for generative neural networks.
• 3D Gaussian SplattingA model that builds a 3D reconstruction from multiple images of a scene along with camera parameters.
• Camera ParameterParameters that describe from where a particular image was captured in 3D space.
• MMDiTMultimodal Diffusion Transformer: a neural network that performs the generative steps.
• Sparse Point CloudA 3D point cloud that describes the initial geometry of a scene and can be used as initialization for 3D reconstruction.
• LoRaLow Rank Adaption: A approach to train neural networks with small "adapters" instead of training the whole model.

Team