Sena Korkut, Emin Sadikhov, Zhenzhang Ye, Nikita Araslanov
September 25, 2024
PDFThis research investigates the generalization of neural 3D reconstruction models across synthetic and real-world datasets. Specifically, we examine the gap between these domains using a model called Splatter-Image for 3D re- construction from a single view input. Two models were evaluated: a pre-trained model and a fine-tuned model on a synthetic dataset. While the fine-tuned model outperformed the pre-trained one on the synthetic test set due to the fine-tuned model’s inclinations towards generating objects with more depth, no improvements were observed on a real-world indoor scene dataset. Although our metrics suggested improvements for the fine-tuned model on synthetic data, the visual results did not fully support the metrics. We analyzed these models on real-world data, and our analysis confirmed the representation differences between synthetic and real-world datasets, highlighting the challenge of the domain gap. The difficulties of using real-world data are clearly demonstrated in this work. We provided insights into the limitations of sim-to-real generalization in 3D object reconstruction and suggest possible future directions to improve model generalization.