From six seconds of generated video to a 3D map, one stage at a time

SLAM (simultaneous localisation and mapping) answers two questions from a moving camera at once: where was the camera at every moment, and what does the world around it look like? Classical systems such as ORB-SLAM answer them by matching keypoints between frames and refining poses and 3D points together in an iterative optimisation called bundle adjustment.

AMB3R-SLAM takes a different route. Its core is a feed-forward 3D model, Depth Anything 3: hand it a batch of images and, in a single forward pass with no iterations, it returns where each camera was, how far away every pixel is, and how confident it is. The catch is that one pass only fits a few dozen frames. The system's contribution is everything around the model: a light front-end that tracks every frame, and a hierarchical backend that stitches hundreds of passes into one consistent map. These are its eight stages.

We wanted footage from a small robot's point of view in an industrial setting, and no robot to film it with. Seedance 2.5 generated it from a prompt: a camera 30 cm above the ground rolling down a dirt lane between stacked wrecks, curving left past tyres and a pickup truck, then past a car with its hood open. One generation at 720p, 6 s long, took 178 s on the provider's side, and we used the first result.

The video is 1280 × 720 at 24 fps: 145 frames. Nothing else goes into SLAM: no camera calibration, no depth, no motion sensor.

Depth Anything 3 is a vision transformer: it cuts each image into 14 × 14-pixel patches and reasons about those. AMB3R-SLAM resizes every frame so the long side is 518 pixels, here 518 × 294, a grid of 37 × 21 patches. Fine detail below a patch is blurred into its neighbourhood; this is why the map is soft at small scales.

The frame rate sets how many frames each submap spans. At 10 Hz or more, the backend takes about one frame in four; a 24 fps clip is well above that.

For each new frame, the small model DA3-Small sees four images together: the current keyframe, the two most recent frames and the new one. It returns their relative poses; the front-end fits a Sim(3) transform, with its scale taken from the keyframe's depth, and that gives the new frame's pose. It ran 144 times, once per frame after the first, for 3.99 s of model time.

A keyframe is the reference the tracker leans on. When less than 35% of the keyframe is still visible in the new frame, or 32 frames have passed, the new frame becomes the keyframe; that happened at frames 0, 17, 47, 61 and 78. And every 32 frames the backend hands the front-end a fresh keyframe: its own newest frame, with its depth.

This is where the 3D comes from. Every 32 frames the backend takes the last 96 frames, keeps the front-end's most confident frame in each group of four, adds frames so no gap exceeds four, and hands those 33 to 35 views to DA3-Nested-Giant-Large in one pass. Out come, for every view, a camera pose, a depth for every pixel, a confidence for every pixel, the camera intrinsics and a sky mask. That set, in its own coordinate frame and units, is a submap.

Each pass took 1.79 to 1.92 s on the RTX 4090. Two short warm-up passes (11 and 21 views, at frames 31 and 63) gave the robot a map before the first full window; both were replaced. The model estimated a focal length of about 248 × 239 pixels at the 518 × 294 input, a horizontal field of view of about 93°, which suits a low robot camera.

Every submap has its own origin and its own scale: a single camera cannot tell a small near scene from a large far one. But consecutive submaps overlap, and the frames they share appear in both. Fitting their two reconstructions of those frames gives a Sim(3) transform (a scale, a rotation and a translation) that places the new submap in the old one's frame. Scale comes from depth ratios; rotation and translation from the shared cameras.

Each new submap is aligned to its predecessor, and to the one before that when they still share frames. The five alignments here shared 18 to 33 frames and fitted with relative residuals of 0.54% to 1.51%, in 0.03 s in total.

Alignment between neighbours accumulates small errors, like a chain. Long-context mapping adds links between submaps that are not neighbours: one extra pass over 24 frames spread across a window of up to eight submaps. If two distant submaps both fit that pass well, the pass links them directly. A link is only kept if the two submaps actually see the same place, which ALIKED keypoints test: at least 20% of their frame pairs must share enough matched keypoints.

Here the window covered all four submaps in one 24-view pass (1.24 s). The only far-apart pair, submaps 0 and 3, failed the test: 1 of 72 frame pairs (1.4%). That is right: the robot's first frames and its last frames look at different parts of the yard.

On long routes, the strongest correction comes from recognising a place seen earlier. Every fifth frame the backend adds the image to a DBoW2 bag-of-words database (ORB features against a fixed vocabulary) and asks for the most similar earlier frame at least 100 frames back. A candidate is then verified by reconstructing both places together in one pass, and checking that they overlap and fit their own submaps before it becomes a loop edge.

This clip has 29 probes and no revisits: the robot never comes back, so no candidate was proposed. On the 13-minute Oxford walk in chapter 12, the same stage closed one loop.

Each submap becomes a node; each alignment, long-context link and loop becomes an edge that says where one node should sit relative to another. A robust Sim(3) optimisation then moves the nodes until the edges agree as well as they can. With loops, this is where accumulated drift is corrected.

Here the graph has 4 nodes and 5 edges and no loops, and the chain of alignments already agreed: the optimisation moved no submap by more than 0.07% of the path length, or changed its scale by more than 0.78%, in 0.12 s.

Each submap pass predicted depth for every pixel. To build the map, the recorder takes one frame in four, prefers the submap where that frame sits nearest the middle, and drops three kinds of pixel: those below the 20th percentile of the frame's confidence, those on a depth edge (a jump of more than 3% within 3 × 3 pixels, where foreground and background blur together), and sky. The rest are placed in the world with their submap's final Sim(3).

From 37 frames that gives 3,330,429 points, and 212,818 when sampling every fourth pixel; exporting took 0.18 s.

The viewer replays the run. By default it follows behind the SLAM camera: the yellow frustum is the pose SLAM computed, with the frame it was looking at on its image plane. Every recorded stage is a layer you can switch on: the front-end's four-view window, keyframes, the latest submap pass, the pose graph, loop probes, and the front-end's live estimate. Colour the points by photo, by submap, by confidence or by time. Reveal them as the camera films them, or only once the backend has committed their submap. Ego view looks through the computed camera, with a slider to blend the real frame over the map.

The whole run took 20.62 s for 145 frames, 7.0 frames per second, after 11.5 s to load the models. The two models' passes took 14.3 s of it. Throughput, not the video's frame rate, decides whether this is real time: at about 6 to 7 frames per second it keeps up with a camera sampled at 5 Hz, and falls behind a 24 fps clip by 3.4×.

Before this project we ran the authors' code on a real recording, for scale: the 13-minute Christ Church walk from the Oxford Spires dataset (2024), one camera of a handheld rig at 5 Hz, 816 m on foot, with laser-scanned ground truth. The tiles are the authors' code; the live estimate on the same walk, before and after our fix, is in chapter 4.

The tracer (src/world_sandbox/slam/trace.py) wraps the methods below without editing them, and saves what each returns; the two fixes we made to the code itself (chapter 4) are in our fork. That record is the contract for replacing a stage: a new front-end, model or retrieval method can be run on the same video and compared stage by stage, in the same figures and the same viewer.