MuViSeg

Object-level mapping and navigation require correspondences between whole instance segments rather than pixels or keypoints. We study how frozen 3D foundation features should be processed for segment matching and introduce MuViSeg: a learned pairwise attention head on MASt3R, multi-scale VGGT feature fusion, and a joint multi-view head that reasons over segments from several views to recover transitive matches.

In zero-shot evaluation on Replica and Virtual KITTI 2 across 0°–180° viewpoint changes, the MASt3R head improves over a Sinkhorn matcher by 4.85 and 25.9 AUPRC points, respectively. The multi-view head is strongest among our heads at the widest Replica baselines, although Sinkhorn remains competitive. Across 102 HM3D navigation episodes, direct segment matchers and sparse keypoint aggregation are statistically indistinguishable, while RoMa's dense matches voted into masks score 13.72–17.98 SPL points lower than direct segment matching.

Keywords   Segment matching · 3D foundation models · Multi-view correspondence · Topological navigation

What is the most effective way to process segments for matching?

A robot revisiting a room has to decide whether this chair seen now is the same chair seen a minute ago. Sparse and dense matchers answer at the wrong level: a downstream system must recover object identity by voting matches inside a mask, and the associations are only as reliable as that voting rule. We take the segment-level route and test three concrete pipelines, each a different answer to the question above.

• H1Matching capacity is better spent in a learned cross-segment head than before the matcher. Supported at small-to-moderate baselines and outdoors; it reverses past 135° indoors.
• H2Preserving multi-scale spatial detail up to the mask boundary yields sharper per-segment descriptors than late pooling. Supported: DPT fusion gives +2.8 AUPRC over single-layer pooling on ScanNet++ validation.
• H3Matching should not be done one pair at a time: segments from several views should be scored jointly. The method we propose. Strongest among our heads exactly where the baseline is widest.

One template, three heads.

A frozen backbone produces per-patch features, a class-agnostic segmenter partitions each image into instance masks, masked average pooling gives one descriptor per segment, and a head refines those descriptors and predicts correspondences. The three variants differ in where capacity sits, which backbone feeds it, and how much context the head sees.

Why the backbone choice is forced

MASt3R's features are pair-dependent: they cannot be computed without the second image, which suits a strong pairwise head but cannot be shared across a variable set of views. VGGT's Aggregator processes all input views in one shared sequence and its features are pair-independent. Joint multi-view matching is therefore only reachable on VGGT — the backbone comparison is a consequence of studying that axis, not an independent variable.

Scoring and abstention

From the scaled affinity between refined descriptors, a row-softmax and a column-softmax give two conditional distributions; the DoubleSoftmax score is the log of their product, which is large only when each segment is the other's best match. Not every segment has a counterpart — some are occluded, out of frame, or absent — so each carries a matchability logit, and the score matrix is augmented with a learnable dustbin row and column. A segment is matched only if both its matchability passes 0.5 and its best score exceeds the dustbin.

Training

• DataScanNet++, ground-truth correspondences from projected 3D instance annotations
• SegmenterSAM, class-agnostic; up to M_max = 100 segments per image
• ObjectiveNormalised NLL over ground-truth pairs + binary matchability cross-entropy, λ = 0.3
• EmbeddingD = 128 shared space; τ ≡ 1 for the MASt3R head, learned τ ≤ 1 for the VGGT heads
• OptimiserAdamW, lr 10⁻⁴, weight decay 10⁻⁴, 500 warmup steps, cosine schedule, bf16
• Hardware2 × NVIDIA RTX 5090 with DDP
• Inference173 ms per joint window at N = 4; 290 ms at N = 6 (measured on the robot capture below)

Clamping the learned DoubleSoftmax temperature to τ ≤ 1 cuts the dominant false-dustbin errors by 36% relative, re-allocating the error budget toward harder wrong-match cases. For this head, calibrated scoring rather than richer descriptors is the binding constraint.

The winner depends on the viewpoint regime.

All models are trained only on ScanNet++ and evaluated zero-shot on Replica (indoor) and Virtual KITTI 2 (outdoor driving). Pairs are stratified into four bins by relative camera rotation. The headline metric is AUPRC; R@1 and R@5 are query-weighted across bins.

In-domain check on real captured imagery

Replica and Virtual KITTI 2 are both rendered. We therefore also report ScanNet++ validation on real captured imagery from held-out scenes — indoor and in-domain rather than zero-shot.

Table 3 — ScanNet++ validation, held-out scenes. All three of our heads sit at or above the Sinkhorn baseline when the domain matches training.

Extra views pay off only where overlap is scarce.

The Joint model is trained at N = 4 and evaluated at N ∈ {2, 4, 6, 8} without retraining. Position-embedding indices 4–7 are never directly supervised, yet the model extrapolates without collapse — consistent with joint attention learning view-relative rather than view-absolute patterns.

Identity held across a moving camera.

The Joint model, trained on ScanNet++ at N = 4, applied without retraining to a sequence captured by a robot at Skoltech. A sliding window of consecutive frames is matched in one joint pass over 60 windows; one colour per instance, kept across all views the model matched it in. Masks are FastSAM proposals rather than the ground-truth masks used in the benchmarks above, and the capture has no ground-truth identities — so this is qualitative evidence, and it is also the only setting here where the segmenter is the one a deployed system would actually run.

Does better matching move the robot?

Each matcher replaces the RoboHop online localizer for HM3D Instance Image Navigation, without retraining. The agent receives a reference image of a single object instance and must reach within 1 m geodesic distance. Every matcher sees the same episode list, map, masks and localizer settings — only the online matcher differs.

What this evidence does not cover.

• Training uses indoor ScanNet++ only; an outdoor source may close the Virtual KITTI 2 gap.
• Quantitative real-image evaluation is indoor and in-domain (ScanNet++ validation), not zero-shot. The in-house capture in Fig. 4 is qualitative, with no ground-truth identities. Real outdoor transfer remains untested.
• Matching quality is bounded by the supplied masks. A segmenter-sensitivity study across SAM granularity and FastSAM is left to future work.
• Closed-loop evaluation is confined to simulated indoor navigation.

Future work should test outdoor training data and extreme-rotation sampling, routing between pairwise and joint heads using deployment cues, real outdoor zero-shot transfer, and sensitivity to segmenter choice and granularity.

BibTeX

@inproceedings{fatykhoph2026muviseg,
  title     = {MuViSeg: Multi-View Segment Correspondences from Dense Geometry Priors},
  author    = {Fatykhoph, Denis and Akhtyamov, Timur and Pakulev, Konstantin
               and Devchich, German and Ferrer, Gonzalo},
  booktitle = {Asian Conference on Computer Vision (ACCV)},
  year      = {2026}
}

The authors used Anthropic's Claude to assist with code development. All content was reviewed and verified by the authors, who take full responsibility for the final work.