Not how high a robot scores.How much it drops on new ground.
A cross-embodiment benchmark for embodied mobility. It tests navigation, spatial sense, locomotion and navigation–manipulation coupling as one capability, across robot bodies, and reports how far performance falls in unseen scenes, on unseen tasks and on a robot it never trained on.
The specification is frozen and the metric code is unit-tested. The reference implementation and scores are paused: the repository was last updated in June 2026, and no model numbers have been published. Details and roadmap.
Robots are weakest exactly where nobody is testing them
Embodied-AI benchmarks each measure a slice of mobility, usually on one robot body, usually reporting peak in-distribution scores. The places where models fail in deployment, such as new buildings, new bodies, rough terrain and moving while manipulating, are the places no benchmark pulls forward.
Slices, not the whole
Manipulation suites treat navigation as one atomic skill. Navigation suites have no arms and no real locomotion. Locomotion is judged inside training frameworks with no held-out protocol.
One body at a time
Cross-embodiment exists as models and as a manipulation taxonomy. No mobility benchmark holds out a robot body and asks a policy to drive it zero-shot.
Peaks, not drops
A model that scores 90% in its training homes and 50% in a new one is, for any deployment, a 50% model. Leaderboards mostly report the 90.
Where the field stands
| Capability | Maturity | What the literature shows |
|---|---|---|
| Quadruped velocity tracking | Solved | Trains in minutes in IsaacLab; robust sim-to-real. |
| Humanoid locomotion | Trainable, fragile | Walks in simulation; large sim-to-real gap. |
| Tabletop manipulation | Fast progress | Long horizons still weak: about a quarter of full household sequences succeed at best. |
| Navigate, then manipulate | Usable, unreliable | Strong language following in unseen homes; zero-shot stacks near 58%. |
| Navigation | Brittle in the wild | Discrete instruction following above 80%; continuous, off-road and social navigation far from solved. |
| Moving while manipulating | Research only | Coordinated fridge-opening class tasks around 20%. |
| Following a person off-road | Near-empty | Trail locomotion is mature; following a target across it is barely studied. |
Five dimensions of moving through the world
Each dimension has frozen formulas in the specification and a unit-tested reference implementation. A composite Mobility Score aggregates them, but the headline figures are the generalization gaps.
D1 · Navigation
Reach a goal. Success rate, SPL, navigation error, nDTW path fidelity, collisions and interventions.
D2 · Spatial sense
Understand and remember 3D space. Cross-room recall, relative spatial reasoning, object permanence under occlusion, map consistency.
D3 · Locomotion
Traverse terrain. Velocity-tracking error, terrain-level success, cost of transport, fall rate.
D4 · Coupling
Hand off between base and arm, or move while manipulating. Base-placement success, robustness to base-pose error, and CNI.
D5 · Joint task
Finish a long multi-step mobile manipulation task end to end, with partial credit weighted by sub-goal.
Most discriminative
D2, D4 and the held-out-body gap are where every current model class is weak. D3 on flat ground is a sanity floor rather than a differentiator.
Measure the drop, and measure whether coordination is needed
Generalization gap
Reported for each dimension along four axes: scene, task, embodiment and domain. Held-out splits are generated deterministically from seeds. The embodiment axis is the strongest probe: train on N−1 bodies, evaluate zero-shot on the N-th. A robust policy has small gaps, not just a high in-distribution number.
Coupling Necessity Index (CNI)
The sequential score comes from a strong navigate-then-manipulate baseline that picks a manipulation-feasible base pose, not a strawman. A high CNI marks a task that truly needs whole-body coordination; near zero means a cheaper decoupled stack is enough. It is a model metric and a property of the task.
One action contract, many bodies
Policies speak a single intent interface. Per-body adapters translate it: legged bodies hand velocity to a low-level RL controller, wheeled bodies drive directly and solve arm IK. A body that cannot do something says so instead of failing silently.
Every ingredient exists somewhere. The combination does not.
coveredpartialabsent
| Benchmark or system | Nav | Spatial | Locomotion | Coupling measured | Held-out body | Gap as headline |
|---|---|---|---|---|---|---|
| Habitat 1/2/3 | kinematic | |||||
| VLN (R2R, RxR, VLN-CE) | scene only | |||||
| BEHAVIOR-1K / BRS | wheeled | |||||
| RoboCasa | atomic | |||||
| ManiSkill3 | ||||||
| Verti-Bench | off-road | |||||
| CrossFormer / NavFoM | model, not benchmark | |||||
| STAR-Gen | manipulation | |||||
| ESI-Bench | ||||||
| Mobility Bench | CNI |
To our knowledge no existing benchmark combines joint-level locomotion, navigation and manipulation across locomotion morphologies in one harness, with a held-out body and a coupling metric. We welcome pointers to work we missed; the full positioning with citations is in the repository.
Which models it puts under pressure
Mobile-manipulation VLAs
D1, D4, D5 and the held-out-body gap. Does base and arm coordination really happen, and does it survive an unseen home?
Manipulation VLAs on a moving base
D4 and D5. Do fixed-base skills transfer once the base moves, and when is tight coordination worth it?
Navigation foundation models
D1 with scene, terrain and body gaps. Where do they collapse off-road, under adversarial layouts, or on a new body?
Language navigation agents
D1 with language goals, plus D2. Spatial grounding, cross-room memory, and continuous-control generalization.
CNI tells you per task whether to invest in whole-body coordination or ship a decoupled navigate-then-manipulate stack. The gaps tell you whether a model that looks good in a demo will survive a new building, a new task composition or a new robot.
Design frozen, harness tested, no scores yet
The specification is frozen. Metrics, splits, the action contract, task logic, the strong sequential baseline and the evaluation runner are pure Python with 200+ unit tests and no simulator needed. Simulator loops and learned baselines need a GPU environment and are clearly marked as blocked. We will not publish model numbers until they come from real runs.
Looking for GPUs, robots and critics
The next step needs IsaacLab runs on a GPU, robot teams willing to add a body, and researchers who will tell us which benchmark we overlooked. The name is a working title. Apache-2.0.