Enhanced Monocular 3D Object Reconstruction

01

Summary

We tried three changes to Splatter Image and compared each one against a baseline trained for the same 30k iterations. Two of them helped. DINO embeddings improved novel-view quality, and depth information helped across the board, most of all when it came from a stronger depth model. A Total Variation loss gave a tiny gain. A Sobel edge loss broke training.

Semantic · DINO

+0.31 dB

Novel-view PSNR, 20.95 to 21.26, with embeddings injected at the bottleneck.

Depth · cross-attention

−14% LPIPS

Novel-view LPIPS falls from 0.240 to 0.206 with MiDaS-Small or DPT-Large depth.

Loss · Total Variation

+0.24 dB

A slight gain in novel-view PSNR. SSIM and LPIPS are unchanged within noise.

Loss · Edge

−8.4 dB

Novel-view PSNR collapses to 12.52. Preserving edges directly does not suit Gaussian splatting.

02

Background

Splatter Image predicts one colored 3D Gaussian for every pixel of the input image, using a single U-Net pass. That makes it extremely fast, and it is competitive with NeRF-based approaches such as PixelNeRF.

Its weakness is shared with those predecessors. Single-view reconstruction is ill-posed, so networks tend to fall back on classification and retrieval instead of truly reconstructing, and they score well on PSNR and SSIM while producing "blobby" averages of training shapes.Tatarchenko et al. (2019) showed that single-view reconstruction networks often behave like retrieval systems. Our hypothesis was that the network is underconstrained. If it is given richer signals, it no longer has to spend capacity guessing what the object is or how far away it sits.

03

Approach

We kept the Splatter Image architecture and changed three things around it: what the bottleneck sees, what the input contains, and what the loss rewards.

Semantic conditioning

We inject DINO or CLIP embeddings of the input image into the bottleneck. The aim is to supply high-level object semantics up front, so the rest of the encoder can concentrate on local geometry and appearance, and to make the representation more consistent across viewpoints.

Figure 1. Semantic conditioning of image. CLIP and DINO embeddings are extracted from the input image and injected into the bottleneck layer of the U-Net.

Depth conditioning

Each Gaussian sits at a predicted depth along its camera ray plus a learned 3D offset, so Splatter Image can be read as an extension of a depth network. But dedicated depth estimators are trained on abundant RGB-D data and are far more robust than a shape estimate learned from scarce 3D data. We therefore add depth in two ways: as a fourth input channel, and through cross-attention between image features F and depth features D at the bottleneck.

Attention(F, D) = softmax( (F·W_Q)(D·W_K)ᵀ / √d_k ) · D·W_V

The three projections W_Q, W_K and W_V are learnable convolutional layers, and d_k is the key dimension.

Loss modification

Total Variation loss penalizes intensity changes along both image axes and suppresses noise. Edge loss runs a Sobel operator over the render and penalizes differences in gradient magnitude, aiming for sharper structure. Both terms were weighted at 0.01, the best value in our sweep.

04

Results

We report PSNR, SSIM and LPIPS on two sets of views.PSNR and SSIM reward pixel-level agreement. LPIPS compares deep features and tracks perceptual similarity better. Higher is better for the first two, lower for LPIPS. Every variant was trained for 30k iterations because of compute limits, so we retrained the Splatter Image baseline with the same budget.The published model was trained for 800k iterations and scores higher. All gains on this page are measured against our 30k baseline, not against that number.

Baseline

At 800k iterations Splatter Image outperforms PixelNeRF. Under our 30k budget it scores lower than both (20.95 against 21.76 PSNR for PixelNeRF), and this retrained model is the number every variant below is compared against.

Table 1. Baseline novel-view performance. Our 30k-iteration retrain is the reference for all later tables. Best values in each column are in bold.

Method	PSNR ↑	SSIM ↑	LPIPS ↓
PixelNeRF	21.76	0.78	0.203
Splatter Image (800k iterations)	21.80	0.80	0.150
Splatter Image (30k iterations)	20.95	0.773	0.240

DINO helps, CLIP does not

DINO beats CLIP on every metric, most clearly on novel views. CLIP ends up essentially level with the baseline (20.91 against 20.95 PSNR). A plausible reason is that CLIP, trained contrastively on image and caption pairs, yields global embeddings, whereas DINO's self-distillation yields more local, object-centric features that help predict unseen surfaces.This is our interpretation. We did not run an experiment that isolates the cause.

Table 2. CLIP-based and DINO-based embeddings, on conditioned and novel views. Best values in each column are in bold.

Method	Conditioned views			Novel views
Method	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
DINO-based	32.21	0.961	0.072	21.26	0.786	0.221
CLIP-based	31.72	0.959	0.077	20.91	0.775	0.230

Depth: the estimate matters more than the plumbing

Cross-attention gave the best conditioned-view numbers overall, reaching 32.77 PSNR and 0.060 LPIPS with MiDaS-Small. As a plain input channel, the depth model made a large difference. DPT-Large reached 21.12 novel-view PSNR and 0.200 LPIPS, while DPT-Hybrid fell to 20.23 and 0.250. Better depth gave better geometry, and the gains were most visible on perceptual quality.

Table 3. Depth integration with different depth models, on conditioned and novel views. Input: depth as an extra channel. X-Attn: cross-attention at the bottleneck. Best values in each column are in bold.

Method	Depth model	Conditioned views			Novel views
Method	Depth model	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
Input	MiDaS-Small	30.17	0.938	0.111	20.76	0.776	0.230
	DPT-Hybrid	28.77	0.929	0.126	20.23	0.766	0.250
	DPT-Large	32.36	0.962	0.083	21.12	0.789	0.200
X-Attn	MiDaS-Small	32.77	0.966	0.060	21.11	0.787	0.206
	DPT-Large	32.71	0.966	0.059	21.15	0.788	0.206

Total Variation edged ahead of the baseline. Edge loss collapsed, dropping to 12.52 PSNR on novel views.

Losses: marginal at best

Total Variation was the best of the loss variants but only slightly beat the original. The edge term hurt badly in every metric, which suggests that explicitly preserving edges does not suit a Gaussian splatting representation.

Table 4. Total Variation loss and Edge loss added to the main loss, on conditioned and novel views. Best values in each column are in bold.

Method	Conditioned views			Novel views
Method	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
Total Variation loss	32.31	0.960	0.079	21.19	0.773	0.243
Edge loss	13.61	0.743	0.297	12.52	0.683	0.342

05

Conclusion

Semantics. DINO embeddings improve generalization to novel views. CLIP embeddings do not.
Depth. Monocular depth constrains the solution space, and better depth models give better reconstructions. Cross-attention is the most reliable way to use it.
Losses. Total Variation gives a slight gain. Edge loss is harmful.

The absolute gains are modest and come from short 30k-iteration runs. The report does not include repeated runs, so very small gaps, such as the Total Variation result, should be read with caution. We see the semantic and depth results as promising directions rather than final numbers.

For future work we would look at pre-training strategies that build geometric understanding before reconstruction, and at ideas from generative modeling, such as mixture-of-experts and autoregressive methods, to improve view consistency.

06

Cite

@techreport{deryo2025enhanced, title = {Enhanced Monocular 3D Object Reconstruction}, author = {Deryło, Adam and Gwiazda, Małgorzata and Sadikhov, Emin}, institution = {Technical University of Munich}, year = {2025} }