Mobility Bench · Open source · Spec frozen

Not how high a robot scores.How much it drops on new ground.

A cross-embodiment benchmark for embodied mobility. It tests navigation, spatial sense, locomotion and navigation–manipulation coupling as one capability, across robot bodies, and reports how far performance falls in unseen scenes, on unseen tasks and on a robot it never trained on.

Status

The specification is frozen and the metric code is unit-tested. The reference implementation and scores are paused: the repository was last updated in June 2026, and no model numbers have been published. Details and roadmap.

The gap

Robots are weakest exactly where nobody is testing them

Embodied-AI benchmarks each measure a slice of mobility, usually on one robot body, usually reporting peak in-distribution scores. The places where models fail in deployment, such as new buildings, new bodies, rough terrain and moving while manipulating, are the places no benchmark pulls forward.

Slices, not the whole

Manipulation suites treat navigation as one atomic skill. Navigation suites have no arms and no real locomotion. Locomotion is judged inside training frameworks with no held-out protocol.

One body at a time

Cross-embodiment exists as models and as a manipulation taxonomy. No mobility benchmark holds out a robot body and asks a policy to drive it zero-shot.

Peaks, not drops

A model that scores 90% in its training homes and 50% in a new one is, for any deployment, a 50% model. Leaderboards mostly report the 90.

Where the field stands

CapabilityMaturityWhat the literature shows
Quadruped velocity trackingSolvedTrains in minutes in IsaacLab; robust sim-to-real.
Humanoid locomotionTrainable, fragileWalks in simulation; large sim-to-real gap.
Tabletop manipulationFast progressLong horizons still weak: about a quarter of full household sequences succeed at best.
Navigate, then manipulateUsable, unreliableStrong language following in unseen homes; zero-shot stacks near 58%.
NavigationBrittle in the wildDiscrete instruction following above 80%; continuous, off-road and social navigation far from solved.
Moving while manipulatingResearch onlyCoordinated fridge-opening class tasks around 20%.
Following a person off-roadNear-emptyTrail locomotion is mature; following a target across it is barely studied.
What it measures

Five dimensions of moving through the world

Each dimension has frozen formulas in the specification and a unit-tested reference implementation. A composite Mobility Score aggregates them, but the headline figures are the generalization gaps.

D1 · Navigation

Reach a goal. Success rate, SPL, navigation error, nDTW path fidelity, collisions and interventions.

D2 · Spatial sense

Understand and remember 3D space. Cross-room recall, relative spatial reasoning, object permanence under occlusion, map consistency.

D3 · Locomotion

Traverse terrain. Velocity-tracking error, terrain-level success, cost of transport, fall rate.

D4 · Coupling

Hand off between base and arm, or move while manipulating. Base-placement success, robustness to base-pose error, and CNI.

D5 · Joint task

Finish a long multi-step mobile manipulation task end to end, with partial credit weighted by sub-goal.

Most discriminative

D2, D4 and the held-out-body gap are where every current model class is weak. D3 on flat ground is a sanity floor rather than a differentiator.

Two design ideas

Measure the drop, and measure whether coordination is needed

Generalization gap

gap(d, a) = score_d(in-dist) − score_d(held-out_a)

Reported for each dimension along four axes: scene, task, embodiment and domain. Held-out splits are generated deterministically from seeds. The embodiment axis is the strongest probe: train on N−1 bodies, evaluate zero-shot on the N-th. A robust policy has small gaps, not just a high in-distribution number.

Coupling Necessity Index (CNI)

CNI(t) = SR_wholebody(t) − SR_sequential(t)

The sequential score comes from a strong navigate-then-manipulate baseline that picks a manipulation-feasible base pose, not a strawman. A high CNI marks a task that truly needs whole-body coordination; near zero means a cheaper decoupled stack is enough. It is a model metric and a property of the task.

18 v0 tasks, spread across the coupling spectrum so CNI has range to measure CNI ≈ 0 · a decoupled pipeline is enough CNI high · coordination is required Pure navigation point goal, language goal, following a moving target Cross-terrain flat, stairs, off-road traversal Decoupleable cross-room pick and place, press a button Long horizon "tidy the kitchen": navigate, manipulate × N, remember Must couple open a door, fridge or drawer; wipe; push with force
Cross-embodiment

One action contract, many bodies

Policies speak a single intent interface. Per-body adapters translate it: legged bodies hand velocity to a low-level RL controller, wheeled bodies drive directly and solve arm IK. A body that cannot do something says so instead of failing silently.

Policy under test VLA, navigation model, or a full stack MobilityAction base twist (vx, vy, ω) end-effector pose target gripper + control-mode tag one contract for every body Wheeled arm · v0 differential drive + IK Quadruped (Go2-class) · v0 velocity to RL controller Humanoid · roadmap held-out body in v2 Drone · later navigation subset Wheelchair · later safety-critical, social Held-out protocol train N−1 bodies, test the N-th
Positioning

Every ingredient exists somewhere. The combination does not.

coveredpartialabsent

Benchmark or systemNavSpatialLocomotionCoupling measuredHeld-out bodyGap as headline
Habitat 1/2/3kinematic
VLN (R2R, RxR, VLN-CE)scene only
BEHAVIOR-1K / BRSwheeled
RoboCasaatomic
ManiSkill3
Verti-Benchoff-road
CrossFormer / NavFoMmodel, not benchmark
STAR-Genmanipulation
ESI-Bench
Mobility BenchCNI

To our knowledge no existing benchmark combines joint-level locomotion, navigation and manipulation across locomotion morphologies in one harness, with a held-out body and a coupling metric. We welcome pointers to work we missed; the full positioning with citations is in the repository.

Who it is for

Which models it puts under pressure

Mobile-manipulation VLAs

D1, D4, D5 and the held-out-body gap. Does base and arm coordination really happen, and does it survive an unseen home?

Manipulation VLAs on a moving base

D4 and D5. Do fixed-base skills transfer once the base moves, and when is tight coordination worth it?

Navigation foundation models

D1 with scene, terrain and body gaps. Where do they collapse off-road, under adversarial layouts, or on a new body?

Language navigation agents

D1 with language goals, plus D2. Spatial grounding, cross-room memory, and continuous-control generalization.

For practitioners

CNI tells you per task whether to invest in whole-body coordination or ship a decoupled navigate-then-manipulate stack. The gaps tell you whether a model that looks good in a demo will survive a new building, a new task composition or a new robot.

Status

Design frozen, harness tested, no scores yet

The specification is frozen. Metrics, splits, the action contract, task logic, the strong sequential baseline and the evaluation runner are pure Python with 200+ unit tests and no simulator needed. Simulator loops and learned baselines need a GPU environment and are clearly marked as blocked. We will not publish model numbers until they come from real runs.

v0Wheeled and quadruped bodies in simulation; spec and harness done
v1Coupling tasks, CNI and joint tasks with live runs
v2Held-out body protocol; humanoid added
v3Real-world validation subset
LaterDrone, wheelchair, public leaderboard
Get involved

Looking for GPUs, robots and critics

The next step needs IsaacLab runs on a GPU, robot teams willing to add a body, and researchers who will tell us which benchmark we overlooked. The name is a working title. Apache-2.0.