09 / 12 · Connect a learned model

Trace one learned number all the way to a pixel

Connect capture, optimization, export and runtime without confusing a viewer with a trainer.

The viewer is the last half of an inverse-rendering loop. To build it correctly, follow one number—such as opacity logit—from an optimizer update into a file, through activation, and finally into blending.

1. Observations define the problem

Capture supplies images, camera intrinsics, camera poses and timestamps. Split training observations from held-out evaluation views/times. Check units and camera conventions before optimizing: wrong poses can look like a model that refuses to learn. Resizing an image also requires scaling focal lengths and principal point coordinates.

2. Differentiate a predicted image

Initialize model parameters. Render the sampled camera/time, compare predicted pixels with target pixels, differentiate the chosen loss and update parameters. Repeat with a defined sampling schedule. A differentiable training rasterizer needs gradients through projection, appearance and composition; this real-time Rust viewer implements forward rendering only.

The existing training laboratory actually updates three parameters of one Gaussian against fixed targets. Use it to watch loss, gradient and candidate update separately. It is a small synthetic learning problem, not full-scene training; the included Python toy trainer provides a separate PyTorch/Adam path.

\widehat I=R(\theta,V,P,t),\qquad L=\|\widehat I-I^*\|^2,\qquad\theta'=\theta-\eta\nabla_\theta L

3. Export the meaning of the learned values

For one opacity logit β, the trainer updates β, the exporter writes β, the decoder reads β, and the shader computes sigmoid(β). Exporting sigmoid(β) into a field the viewer activates again changes the model. Apply the same audit to log-scales, quaternion order, motion time units and SH coefficient ordering.

A training checkpoint may contain optimizer moments and neural network weights. A runtime package contains whatever its forward path needs. A deformation-based 4DGS model can require a deformation network; a native 4D Gaussian requires conditioning its joint spacetime distribution. Renaming either file to STG-Lite cannot make our polynomial decoder correct.

4. Validate the bridge at three levels

First compare decoded scalar values with the trainer’s export. Next compare a few projected centers and covariance matrices under a known camera. Finally compare the rendered image at that camera and timestamp. A beautiful orbiting preview can hide an activation or coordinate mismatch that a small numeric check immediately reveals.

RUN → OBSERVE

Make this step concrete

You can point to the exact same raw opacity value in the training record, exported bytes and runtime decode, then explain its effect on a pixel.

Open GPU milestone 7 →

npm run qa:math
# Open /math/#training-lab for the step-by-step optimizer.
# Open /learn/12-build-and-explain/ for the Python training/export capstone.

Commands use the reference repository. Rust/WGSL changes need a native rebuild or npm run build:wasm, followed by a browser reload.

IDEA → IMPLEMENTATION

Follow the running code

Excerpts are extracted from the repository at site build time. Expand only the part you are studying; full files include context and line numbers.

src/workshop.wgsl · L72–79

The runtime end of the bridge: activated opacity becomes a pixel contribution.

@fragment fn fs_main(input: Out) -> @location(0) vec4<f32> {
    let stage = u32(globals.viewport_stage.z);
    if stage <= 2u { return vec4(input.rgba.rgb, 1.0); }
    let weight = exp(-0.5 * dot(input.local, input.local));
    let alpha = min(0.99, input.rgba.a * weight);
    if alpha < 1.0 / 255.0 { discard; }
    return vec4(input.rgba.rgb * alpha, alpha);
}

Open full source with line numbers ↗

Real-model render pass ↗ · Real-model shader ↗