Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models

Novel view synthesis without a 3D model

TTB takes 16 to 64 posed photos of a scene and renders it from new viewpoints at 960p. No 3D model is built in between: every frame comes from the baked MLP layers alone. Switch between 16, 32 and 64 views to see detail sharpen while the frame rate stays at 23 fps.

16 views 32 views 64 views

Input · 16 of 64 views

camera data unavailable

Output

Live demo

The Hugging Face Space runs the whole pipeline on one GPU: upload 16–128 photos or a short video, π³ estimates the poses, TTB bakes the scene, and the viewer streams freshly rendered frames as you move. The network renders each frame on the GPU.

Motivation

Large view synthesis models, which render novel views without an explicit 3D representation, have caught up with per-scene optimization. Given 64 photos of a DL3DV scene and one minute, a KV-cache model (DVSM) reaches 31.1 dB. 3D Gaussian Splatting reaches 27.3 dB from the same 64 photos, and 30.5 dB when optimized on about 300.

Why the KV-cache slows down

A large view synthesis model reconstructs a scene once, then renders many novel views from that state. The state is not a 3D model: a KV-cache model stores the scene as the keys and values of every input token, and every frame attends to all of it, so each added photo makes every frame slower. TTB instead bakes the scene into fast weights: one small MLP per block, written at test time. Their size is fixed, so more photos change what the weights hold, not how many there are.

Real-time comparison

Both panels fly the same camera path in real time, and each shows the newest frame its model has finished. At 64 views the KV-cache finishes about one frame per second and TTB 23; at 16 views the KV-cache reaches 4 and TTB stays at 23. The images are nearly the same (under 1 dB apart on DL3DV), because TTB starts from the KV-cache model and keeps its cross-view attention for reconstruction.

16 views 32 views 64 views

KV-cache

1 fps

TTB

23 fps

Method

Baking attention into fast weights

TTB builds on Test-Time Training (TTT) layers, which replace attention with a small MLP and train it on the input views at test time. TTT asks the MLP to map each token's key to its own value. TTB keeps cross-view attention running during reconstruction and asks the MLP to reproduce the attention output instead. Once the MLP is trained the KV-cache is dropped, so rendering costs exactly what TTT costs, while the weights hold what attention gathered from all views.

Context pairing

One mismatch remains. During reconstruction the MLP sees tokens that carry image and pose; at render time it is queried with pose only. TTB therefore gives every context view two tokens: a pose-only rendering token that goes into the MLP, and a ground-truth token with the image that feeds the attention target. The MLP is trained on the same kind of input it is later asked about.

The plots under each variant compare reconstruction and rendering tokens at blocks 1, 4 and 7. Only full TTB mixes the two from the first block; the other two keep them apart until late in the network.

Each TTB layer bakes the map $X^{(\text{ctx-rend})} \mapsto \mathrm{CrossAttn}(X^{(\text{ctx-gt})})$ and shares the baked MLP with the novel-view tokens, so both stages see the same modality. Reconstruction handles twice the tokens; rendering cost is unchanged.

Benchmarks

We evaluate on the 140 held-out DL3DV scenes at 960p with 16, 32 and 64 input views. TTB renders every setting at 23 fps, up to 23× faster than the KV-cache model it starts from (DVSM in the table) and 2.3× faster than LaCT, within 1 dB of the KV-cache's quality. With local-duplicate inference, which keeps one copy of the fast weights per 16 views, it passes the KV-cache at 64 views (31.33 vs 31.11 dB). The cost is paid once, at reconstruction: 65 s for 64 views, against 60 s for the KV-cache and 7.3 s for LaCT.

TTB-- uses a linear-projection target instead of attention, so reconstruction stays linear in the number of views; it matches LaCT's quality with fewer parameters and faster rendering.

Zero-shot generalization

TTB was trained on DL3DV only. Given 64 photos of scenes from Mip-NeRF 360, Free and Hike, it renders at the same 23 fps and stays close to the KV-cache model in quality. Hover a scene to play it.

64 input views at 960p.

Ablation

Three additive changes turn a TTT replacement into TTB: context pairing, an MSE inner loop with three steps, and the cross-view attention target. Together they close the gap to the KV-cache model at TTT's rendering speed.

Citation

@article{lin2026ttb,
  title     = {TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models},
  author    = {Lin, Chin-Yang and Hachiuma, Ryo and Choe, Jaesung and Chen, Min-Hung and Wang, Yu-Chiang Frank and Lin, Yen-Yu and Chiu, Wei-Chen and Liu, Yu-Lun and Sun, Cheng},
  journal   = {arXiv preprint arXiv:2611.17331},
  year      = {2026}
}

References

1. Test-Time Training Done Right (LaCT). ICLR 2026. arXiv:2505.23884
2. DVSM: Decoder-only View Synthesis Model Done Right. arXiv 2026. arXiv:2605.29891