OpenReality · Experiment 21Complete · 2026-07-09
Better training data, measured
From ordinary monocular video, OpenReality reconstructs robot-training trajectories that are 8–37× smoother than the incumbent methods — the property that decides how well a policy learns — on every task tested, at p < 1e-6.
- H3 · data qualityDecisive win8–37× smoother trajectories, every task, p < 1e-6
- H1 · accuracyCell-dependentwins moving-camera pick-place, ties insertion, loses close-range stacking
- Arm C · SplaticaVisualization-gradeno trajectory at all, ghosts the moving object, ≈4× the cost
- H2 · absolute scaleHonest negativeno method hits usable absolute metres on this data — one reference fixes it
H3 · data quality
Temporal coherence — the win that holds everywhere
Action jerk (the third derivative of motion; lower is smoother) is what separates training-grade trajectories from noisy ones. On every task, the reconstructed trajectory is far smoother than Encord's 2D-keypoint annotation.
| Task | Dataset | OpenReality | Encord | Result |
|---|---|---|---|---|
| Pick-and-place | DROID · wrist cam | 13.2 | 108.9 | 8.2× smoother |
| Insertion | FMB · wrist cam | 11.7 | 269.6 | 23× smoother |
| Stacking | RH20T · in-hand | 2.85 | 104.5 | 37× smoother |
median action jerk, m/s³ · wrist / in-hand cameras · clean input
Action jerk −106.3 m/s³ (95% CI [−157.8, −66.5]), velocity noise −30.6 m/s², both p < 1e-6. Every arm stays above the encoder-truth smoothness floor — 0 / 24 cells over-smoothed — so it recovers real motion rather than blurring it away.
H1 · accuracy
Position error is cell-dependent — losses included
End-effector position error on moving-camera capture. We report every cell with equal weight; the losses are the point, not a footnote. Pooled across all moving-camera cells it is a statistical tie (−2.6 mm, 95% CI [−11.1, 3.2], p = 0.157).
| Cell | OpenReality | Best baseline | Verdict | Detail |
|---|---|---|---|---|
| Pick-and-placeDROID · moving camera | 34.9 mm | ≈79 mm | win | ≈2.2× tighter · paired −18.5 mm, p ≈ 0.01 |
| InsertionFMB · moving camera | 18.8 mm | 18.6 mm | tie | statistical tie · p ≈ 0.4 |
| StackingRH20T · in-hand, contact-rich | 19.5 mm | 9.6 mm | loss | baseline wins close range · p ≈ 0.07–0.10 |
H4 · robustness
Degradation & occlusion
How error moves under blur, low light, and frame-rate drops — and under occlusion, per cell. Where occlusion has no effect, we say so.
Arm C · Splatica's off-the-shelf 3D Gaussian splatting
Splatica-style off-the-shelf splatting is visualization-grade
Run at its best case (100% registration on every episode), vanilla COLMAP + gsplat still produces no trajectory at all, ghosts the moving object while the static scene stays crisp, and costs ≈4× more per clip ($0.31 vs $0.075). It is a scene-visualization tool, not a manipulation-data tool.
H2 · absolute scale
What this does not claim
On this data, no method delivers usable absolute metres — every arm’s scale error exceeds 85%. The “native-metric” mono-depth baselines are actually worse than uncalibrated units (256% and 170% median scale error). OpenReality ships scale-normalized metres; one disclosed reference measurement is what turns them into true metres. We do not claim absolute metric accuracy here.
- Baselines were tuned as hard as our own pipeline — frozen grid, disjoint split, per-cell locked configs. That is what makes the result credible.
- Accuracy is scoped to moving-camera capture — the losses (close-range stacking) and ties (insertion) are shown, not hidden.
- Public ground truth only — DROID, RH20T, FMB robot-encoder / mocap data. No customer or pilot data appears anywhere in this benchmark.