residential surveyFamily Home
A walk from entryway to kitchen becomes a dense 3D map with seating, appliances, and storage indexed as the camera moves.
Products
Scan the world. Run it in the OS.
Live
Open Reality Scan
The front door: point your phone and reconstruct the space in real time — scene report, spatial agent, and revisitable scans. No rig required.
Early access
Open Reality Operating System
Where captured space becomes a working environment: phone video, a depth-camera capture, or a splat you already have turns into a persisted, splat-native scene. Agents label objects with evidence, measure in metres once anchored, and plan robot-height paths — planner visualization, not certified navigation. Scenes export as LeRobot v2 bundles, with an experimental Isaac Sim lane.
Get in touchClaude Code
Your scans, inside Claude
openreality-mcp puts the whole OS in Claude Code and Claude desktop: upload captures, run reconstruction, measure and query scenes, talk to the scene agent, and pull robot-training exports, all as tools Claude can use.
claude mcp add openreality -- npx -y openreality-mcp serveFull setup guide covers Claude desktop and signing in.
How it works
Raw demo video in, policy-ready episodes out
OpenReality runs every demo through one pipeline: reconstruct it in metric 3D, ground and QA the trajectories, then export clean LeRobot episodes your policies can train on.
- input · capture
Demo video
Your teleop or human-hand demo. Ordinary RGB video, no rig or depth sensor.
mp4 · rgb frames - reconstruct · slam
3D reconstruction
VGGT-SLAM recovers metric, gravity-aligned geometry and per-frame camera pose.
point cloud · SE(3) pose - ground + qa · repair
Grounding + QA
Objects get labeled and anchored in 3D. Trajectories are extracted, checked, and flagged where a take fails.
trajectories · labels · flags - export · dataset
LeRobot dataset
Clean, scale-consistent episodes in LeRobot format, ready to train a policy.
lerobot · episodes
preview · experimental
Experimental / previewIsaac sim preview
Optional preview of episodes replayed in Isaac. Unvalidated, with no sim-to-real claims yet.
usd · sim previewpipeline · monocular video → metric 3D → grounded + QA trajectories → lerobot episodes · isaac preview (experimental)
Before / after
A raw demo in, a trainable episode out
Start from the demonstrations your team already has. OpenReality returns the same episode with metric geometry, trajectories, grounded labels, and quality flags, in LeRobot format your policies train on directly.
beforeRaw teleop / human-hand demo
- RGB framesmp4 · rgb
- Action / state logcsv · raw
afterRefined LeRobot episode
- RGB framesmp4 · rgb
- +Metric depthdepth · per-frame
- +Camera + gripper trajectorySE(3) · per-frame
- +Grounded object labelslabels · 3D anchored
- +Per-frame quality flagsflags · drift · occlusion
Load it like any LeRobot dataset
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
ds = LeRobotDataset("openreality/your-demos-refined")
ep = ds[0]
ep["observation.images.cam"] # rgb frames (kept)
ep["observation.depth"] # metric depth (added)
ep["observation.state"] # SE(3) trajectory (added)
ep["annotations.objects"] # grounded labels (added)
ep["quality.flags"] # failure flags (added)Validation in progressWhat we are measuring next
- Policy liftSame demos, same model, baseline vs refined. Metric = task success and data-efficiency.
- Dataset salvageDemos flagged for drift, occlusion, missing contact, or bad calibration, then repaired or rejected.
Proof · benchmark
The trajectories come out 8–37× smoother
On DROID, FMB and RH20T — public robot datasets with encoder-grade ground truth — motion reconstructed from ordinary monocular video carries far less jerk than Encord's 2D-keypoint annotation on every moving-camera task tested, at p < 1e-6. (The off-the-shelf splatting arm produces no trajectory to compare; fixed-camera cells show a far smaller margin.) Smoothness is the property that decides how well a policy trains.
- H3 · data qualityDecisive win8–37× smoother trajectories, every task, p < 1e-6
- H1 · accuracyCell-dependentwins moving-camera pick-place, ties insertion, loses close-range stacking
- Arm C · SplaticaVisualization-gradeno trajectory at all, ghosts the moving object, ≈4× the cost
- H2 · absolute scaleHonest negativeno method hits usable absolute metres on this data — one reference fixes it
m/s³ · median over episodes
Pooled paired test · 69 episodes
Action jerk −106.3 m/s³ (95% CI [−157.8, −66.5]), velocity noise −30.6 m/s², both p < 1e-6. And every arm stays above the encoder-truth smoothness floor — 0 / 24 cells over-smoothed — so it recovers real motion, it doesn’t blur it away.
Position accuracy is cell-dependent — we report the losses too
End-effector error on moving-camera capture. We win where it counts for monocular capture, tie on insertion, and lose close-range contact-rich stacking. Pooled across every moving-camera cell it’s a statistical tie (−2.6 mm, p = 0.157). The baselines were tuned as hard as our own pipeline.
- winPick-and-placeDROID · moving camera34.9 mmvs≈79 mm≈2.2× tighter · paired −18.5 mm, p ≈ 0.01
- tieInsertionFMB · moving camera18.8 mmvs18.6 mmstatistical tie · p ≈ 0.4
- lossStackingRH20T · in-hand, contact-rich19.5 mmvs9.6 mmbaseline wins close range · p ≈ 0.07–0.10
Splatica's off-the-shelf 3D Gaussian splatting costs ≈4× more per clip ($0.31 vs $0.075), produces no trajectory at all, and ghosts the moving object — visualization-grade, not training-grade. Ground truth: DROID · RH20T · FMB (public robot datasets). No absolute-metric claim — OpenReality ships scale-normalized metres.
Read the full benchmark →Data fidelity
Capture quality you can see
Reconstruction quality is decided at capture time. Move slowly around an object with plenty of overlap and you get dense, complete geometry. A quick casual pass leaves gaps — sparse points, noise, whole regions never seen. Here is the same ceramic mug captured both ways, in true colour. Drag to orbit; turn the casual capture to find the side it never saw.
What to look for
Dense, complete — every side observedSparse, with gaps — a whole side missing
Illustrative demoBoth clouds are the same scanned object in real per-point colour. The casual capture is a simulated degradation — subsampled, noised, and clipped on one side — to show a coverage gap, not two separate field captures.
Casual passquick single sweep
- points
- loading…
- coverage
- partial · one side missing
Deliberate passslow multi-view orbit
- points
- loading…
- coverage
- complete · all sides
Where a casual pass loses coverage
- Low view overlap
- Surfaces seen in too few frames stay sparse and noisy.
- Unvisited angles
- A side you never walk around never gets reconstructed.
- Fast, shaky motion
- Motion blur and weak parallax thin out the cloud.
Deliberate, multi-view capture returns dense, complete geometry you can train on. A casual pass returns sparse, holey clouds — the gaps a policy can't learn from, and the regions a refinement pass has to re-observe before data ships.
Demo object: “Cole Hardware Mug Classic Blue,” Google Scanned Objects, © Google LLC, CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).
Motion QA
Is the motion trainable?
Geometry QA covers a single instant. Robot policies learn from motion, so the same sequence has to hold consistent depth, trajectories, and action boundaries across every frame. These are the signals we validate before motion data is labeled.
Motion QA · depthper-pixel depth · near → far
depth RMSEvalidation in progress
IllustrativeWhy it gates labelingValidates metric scale — depth that drifts off scale poisons the policy.
Depth → metric geometry
Per-pixel depth becomes a metric, gravity-aligned point cloud — the same scaffold geometry QA scores. Depth that drifts off metric scale is caught here, not in the policy.
depth · metric scale
Point tracks → trajectory consistency
Following the same surface points across frames recovers each object's trajectory. Reprojection error tells us which tracks are reliable enough to supervise a policy.
tracks · reprojection error
Motion masks → action labels
Separating what moves from what stays still segments actions over time, so a sequence carries temporal labels — not a single frozen instant.
masks · temporal coverage
4D reconstruction + tracking · depth · point tracks · motion masks · metrics: validation in progress
Grounding engine
The agent that labels and QAs every take
The same agent reasons over the reconstructed 3D scene, not flat frames — anchoring objects in metric space, tracking what moved, and flagging where a take fails. Those grounded labels and quality flags are what ride into each LeRobot episode.
Simulated agent sweepreal Open Reality capture · illustrative anchors
Drag to orbit
The walkthrough above is our own simulated agent sweep over a real Open Reality capture — we demo in simulation, and the anchors and labels are illustrative, not live model output. The same spatial memory is built to run on real robots: OpenClaw · embodied AI for robotics.
Field recordings
Scenes you can explore
These are real captures reconstructed by the live pipeline. Run any of them from your dashboard to replay the full system with mapping, detection, and the spatial agent.
residential survey
facilities surveyResearch Building
One pass through a university building's hallways: workstations, whiteboards, and safety equipment, all tracked in place.
forensic documentationCrime Scene
Evidence markers and disturbed objects documented with their spatial relationships intact with a scene you can re-walk later.
disaster responseFlooded Street
A flood-damaged street surveyed for structural damage, displaced objects, and roadway hazards in a single walk-through.
damage assessmentEarthquake Aftermath
Urban damage assessment: debris fields, failed infrastructure, and blocked pathways mapped from one continuous pass.
event venueHackathon Venue
A live event space mapped in motion with seating, collaborative work areas, and circulation routes captured as they're used.
interior surveySecond Interior
A second residential survey focused on layout and usage patterns with furniture, electronics, and storage, room by room.
team workspaceOur Workspace
The team's own table, scanned mid-build with laptops, tools, and the reconstruction itself running on screen.
Our work
What happens under the hood
Dense SLAM, in real time
A feed-forward transformer predicts dense depth and camera pose straight from the monocular video stream, no LiDAR, no depth sensor. Submaps are stitched on the SL(4) manifold and optimized with GTSAM, so the map stays consistent even when your path crosses itself.
VGGT-SLAM 2.0 · SL(4) · GTSAM · loop closure
Detection without a label set
Tell the system what matters in plain language instead of picking from a fixed list of classes. Open-set detection segments the objects you asked for and anchors them in the 3D map, where they keep their positions as the scan grows.
CLIP · SAM3 · open-set 3D detection
An agent that knows where things are
An autonomous spatial agent connects your goal to the live geometry: it plans what to track, runs deep scans on regions of interest, and answers questions grounded in what the camera has actually seen — during the scan and after it.
autonomous missions · grounded Q&A · scene reports
Any phone, no app
Capture runs in the mobile browser — open a link, point the camera, walk. The heavy lifting happens on a dedicated GPU session in the cloud, streamed back to you as the map assembles.
mobile browser capture · dedicated GPU session
For robotics teams
Build a grounded dataset from real captures
Pilot a capture workflow for your environments, objects, and robot learning format targets.
Begin
Open your reality.
A few minutes of walking is enough to map a space you can revisit, search, and question.