Summary
We took Splatter Image, which reconstructs an object from a single photo, and fine-tuned it on synthetic furniture from 3D-Front. We then compared it with the original model, pre-trained on Objaverse, on both synthetic and real data. Fine-tuning clearly helped on synthetic test data. On real indoor scenes from ScanNet++ it made no meaningful difference, and the reconstructions looked different from what the metrics suggested.
Background
Synthetic data is cheap, clean and comes with exact camera poses, which makes it attractive for training 3D reconstruction models. Real scenes are messier: lighting varies, objects occlude each other, and materials are mixed. The open question is whether what a model learns from synthetic data survives that jump.
We study this with Splatter Image, which predicts one 3D Gaussian for every pixel of a single input image and places each one using a predicted depth plus an offset.Our other project, Enhanced Monocular 3D Object Reconstruction, looks at Splatter Image in more detail. We also wanted to examine the contrastive training ideas from ContraNeRF, which targets synthetic-to-real transfer. The modules it depends on were missing from the code we had, so we focused on Splatter Image alone.
Setup
Datasets
3D-Front is a synthetic collection of furnished rooms and object models. We first hoped to keep the scene noise, such as occlusion and lighting, but we had no camera poses for full scenes, so we used the individual object models instead. ScanNet++ is our real-world set: indoor scans with iPhone video frames and DSLR images, all with camera poses.
Preprocessing
For 3D-Front, we rotated each object randomly to create several views at 128 × 128 pixels and saved each view's camera transformation. For ScanNet++, we projected the 3D semantic labels onto the DSLR images using the camera poses, which gives a mask for every object instance (Figure 1). We used five DSLR images per scene and picked early video frames, where the camera has barely moved, so the views of an object differ only slightly (Table 1). The object classes we extracted were chairs, tables, office chairs, cabinets, bookshelves, sofas, beds, monitors, storage cabinets and doors.
Models
The pre-trained model is Splatter Image trained on Objaverse. The fine-tuned model is the same network further trained on 3D-Front. We also tried training on 3D-Front from scratch, but the dataset is too small to give meaningful results, so we dropped that variant.
Metrics
We follow Splatter Image and report PSNR, SSIM and LPIPS.PSNR and SSIM reward pixel-level agreement. LPIPS compares deep features and tracks perceptual similarity better. Higher is better for the first two, lower for LPIPS. Before evaluation, each object is extracted, its position is normalized, and it is placed on a white background. Images that are not 128 × 128 are resized.
Results
Table numbers follow the report.Table 6 of the report, which shows a blanket and a curtain reconstructed from ScanNet++, is not reproduced here.
Fine-tuning helps on synthetic data
On 3D-Front the fine-tuned model is better on all three metrics, with PSNR rising from 18.906 to 23.439, SSIM from 0.873 to 0.919 and LPIPS falling from 0.136 to 0.121. That is expected, since the model has now seen similar data during training.
| Dataset | Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| 3D-Front | Pre-trained | 18.906 | 0.873 | 0.136 |
| Fine-tuned | 23.439 | 0.919 | 0.121 | |
| ScanNet++ | Pre-trained | 2.843 | 0.230 | 0.449 |
| Fine-tuned | 3.445 | 0.259 | 0.442 |
On real scenes the numbers say little
On ScanNet++ both models score close to the floor (PSNR 2.843 and 3.445). The evaluation views are almost identical to the input view and mostly show the front of each object, so the metrics only test how well the front is reconstructed and say little about the 3D shape. The differences between the two models are small.
The visual differences are larger than the metrics
Tables 3 and 4 show example reconstructions from three views. The pre-trained model tends to produce compact, well-defined objects. The fine-tuned model produces deeper objects that are less clear. We suspect this comes from 3D-Front, where large items such as sofas and beds are common. This difference in appearance is bigger than the metrics suggest.
The real inputs are also noisy. Objects in ScanNet++ have holes where something was placed on top of them or where they were cut off, and both models fill those holes with white, the background color.
Difficult objects
We also scored only the objects that look hard to reconstruct: curtains, blankets and blinds. Results are mixed, with the fine-tuned model ahead on PSNR and SSIM and slightly behind on LPIPS. Visually, the fine-tuned model gives less opaque, deeper reconstructions, whereas the compact shapes of the pre-trained model may suit curtains better. The metrics do not capture this difference.
| Dataset | Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| ScanNet++ | Pre-trained | 2.571 | 0.236 | 0.438 |
| Fine-tuned | 2.975 | 0.253 | 0.441 |
Conclusion
- Synthetic data. Fine-tuning on 3D-Front improves results on synthetic test data.
- Real data. Both models struggle on ScanNet++, so the sim-to-real gap remains.
- Evaluation. The metrics miss differences in 3D quality, especially when the evaluation views are close to the input.
Limitations. PSNR, SSIM and LPIPS say little about 3D quality when only a few similar views are available. The fine-tuned model's reconstructions are sometimes noisier. Good real-world inputs are also hard to obtain.
Future work. Train on full 3D scenes instead of single objects, add contrastive learning in the spirit of ContraNeRF, and change the loss to emphasize general object features over fine detail.