Robot Teleoperation Demonstrations
Robots operated by people, recorded as synced video, robot actions and whether each task succeeded.
- 400K+robot episodes
- 35K+new episodes every month
Sample
One episode and its records
Booster T1 episode, light-box pick-and-place (23-DOF)
Representative of the real metadata shape. Audio is 16 kHz wav; counts re-verified on disk (925 depth frames, 2,338 joint samples).
{
"episode": {
"id": "episode_f0fe8050-...",
"version": "2.0",
"duration_seconds": 29.51,
"task": {
"name": "Light Box Table To Chair Behind V0",
"instruction": "Pick up the light box, turn around, and place it on the chair behind you."
}
},
"timing": {
"start_unix": 1766526819.3,
"fps_target": 30,
"fps_actual": 40.06
},
"sensors": {
"visual": {
"rgb_frames": 1182,
"depth_frames": 925,
"resolution": {
"width": 1280,
"height": 720
}
},
"proprioception": {
"joint_samples": 2338,
"joint_sample_rate_hz": 79.23
},
"audio": {
"format": "wav",
"sample_rate": 16000,
"channels": 1
}
},
"stats": {
"recording_quality": "good",
"active_streams_count": 5
},
"provenance": {
"watermarked": true,
"signature": "TW7wMQxo...A8="
}
}Per-step proprioception frame (23 named joints)
Representative. One timestep; positions in rad, velocities in rad/s, efforts in Nm.
{
"timestamp": 3.0976,
"unix_timestamp": 1766526822.3984,
"positions": {
"Left_Knee_Pitch": 0.4141,
"Right_Knee_Pitch": 0.4229,
"Waist": -0.0048,
"Head_pitch": 0.3603
},
"velocities": {
"Left_Knee_Pitch": -0.00929,
"Right_Knee_Pitch": 0.00759
},
"efforts": {
"Left_Knee_Pitch": -4.2377,
"Right_Knee_Pitch": -5.0484,
"Left_Hip_Yaw": 2.9137
}
}LimX Tron quadruped, fall-recovery, success-labeled
Representative, cross-embodiment. Shows the success label and per-stream rates including 915 Hz motors.
{
"episode": {
"id": "tron_success_001",
"duration_seconds": 15.6
},
"machine": {
"type": "limx_tron1",
"class": "quadruped_robot"
},
"task": {
"type": "fall_recovery_v0",
"success": true,
"failure_reason": null
},
"sensors": {
"feed": {
"proprioception": {
"motors": {
"sample_count": 14277,
"avg_rate_hz": 915.2
},
"imu": {
"sample_count": 1518,
"avg_rate_hz": 97.3
},
"odometry": {
"sample_count": 1501,
"avg_rate_hz": 96.2
}
}
}
},
"files": {
"total_size_bytes": 64957399
}
}Specifications
| Formats | H.264, JSONL, NPZ |
|---|---|
| Delivery | S3 / pre-signed URL, REST API, CDN |
| Cadence | 35K+ new episodes per month |
| Coverage | 2023 to 2026 |
| Episodes | 400K+ accepted |
| Modalities | RGB video + proprioception + IMU |
| Labels | Success + task metadata |
| Platforms | Booster T1, Unitree H1/G1, LimX Tron |
| Sync | <1ms cross-stream |
| Access | Org-scoped API keys, tenant isolation, per-customer storage paths, and mutual TLS. Artifacts are signed, watermarked, and access-logged under a restricted per-delivery license. |
| Built against | Open X-Embodiment, LeRobot, RLDS |
What ships
Manipulation Trajectories
Complete teleoperated task demonstrations with synchronized vision and robot state across diverse manipulation tasks.
State-Action Pairs
Frame-aligned observation and action tuples ready for behavior cloning and inverse RL.
Success Labels
Human-verified task outcomes and metadata for filtering, curriculum design, and reward modeling.
Fields
15 fields
| Field | Description |
|---|---|
| episode.idstring | Unique episode identifier.Example episode_f0fe8050-... |
| duration_secondsfloat64 · s | Episode wall-clock duration.Example 29.51 |
| task.instructionstring | Language-conditioned instruction given to the operator.Example Pick up the light box and place it on the chair behind you. |
| task.successbool · nullable | Human-verified task outcome (null if unevaluated).Example true |
| task.failure_reasonstring · nullable | Reason on failure; null on success.Example null |
| timing.fps_targetint · fps | Target video frame rate.Example 30 |
| timing.fps_actualfloat · fps | Measured achieved frame rate.Example 40.06 |
| sensors.visual.rgb_framesint · frames | RGB frame count (mp4, H.264).Example 1182 |
| sensors.visual.depth_framesint · frames | Depth frame count (16-bit PNG).Example 925 |
| sensors.visual.resolution{w,h} · px | Image resolution.Example 1280 x 720 |
| sensors.proprioception.joint_samplesint · samples | Joint-state sample count.Example 2338 |
| sensors.proprioception.joint_sample_rate_hzfloat64 · Hz | Measured joint sampling rate.Example 79.23 |
| sensors.audio.sample_rateint · Hz | Audio sample rate (wav, mono).Example 16000 |
| license.license_idstring | Per-delivery license id.Example GRR-20251223-A135CB89 |
| provenance.signaturestring | Cryptographic integrity signature; episodes are watermarked.Example TW7wMQxo...A8= |
Load it
The first commands after an approved delivery.
# Load and convert an ORTF episode set GET /datasets # list available datasets GET /datasets/{id} # metadata + episode index GET /datasets/{id}/download # pre-signed URL to the ORTF bundle ortf validate ./dataset # manifest + frame-count + timestamp checks to_lerobot(load_dataset("./ortf"), "out/lerobot") # convert to LeRobot v3.0
Quality checks
3 checksSim2Real Gap Characterization
How faithfully a MuJoCo simulation reproduces the real teleoperated trajectories - which validates that the captured real data is the ground-truth reference.
Position-controlled trajectory replay in MuJoCo over 123 episodes, 3,687 seconds, and 284,794 joint samples, scored per joint with MAE, RMSE, and Pearson correlation plus a composite gap score.
Real measured: aggregate position MAE 5.56 degrees, knee pitch highest at 12.2 degrees, velocity MAE 0.41 rad/s, legs 6.7 vs arms 5.5 degrees (about 1.2x), overall gap score 0.36. See the sim2real paper.
Episode acceptance / QA gate
Whether an episode is clean enough to enter the saleable catalog.
Automated format, stream-presence, and quality checks plus human QA, with a recording-quality tag, an active-stream count, and success/fail bucketing.
Methodology-stage. Pass/fail per episode; no published aggregate acceptance-rate metric.
ORTF schema validation
Whether a packaged delivery conforms to the format.
A validator checks manifest schema, required files, parquet-to-manifest match, video frame-count integrity, timestamp monotonicity, and action/observation dimension match.
Methodology-stage. Validator pass/fail, not a score.
Ground truth
| What is correct | The human teleoperator executed trajectory is the demonstration ground truth, and the human-applied success boolean and failure reason are the task-outcome ground truth. For simulation fidelity, the real robot trajectory is the reference the simulation is scored against. |
|---|---|
| How it is established | Success labels are applied by humans and encoded per episode. Sim2real validation is an execution-based comparison of simulated vs real joint trajectories via per-joint MAE, RMSE, and correlation, with a normalized composite gap score. Integrity is graded by per-file SHA-256 checksums and a signature. |
| Agreement | No inter-rater agreement statistic is published. Correctness for the simulation study is anchored to the recorded real trajectory itself. |
How it is built
- 01
Human teleoperation capture
Episodes are recorded by human teleoperators driving real robots (Booster T1, LimX Tron, Unitree). Each episode is a single contiguous recording with a task name, description, and language instruction.
- 02
Multi-modal synchronized recording
RGB (mp4/H.264), depth (PNG), proprioception (joint position, velocity, effort), robot state, and audio (wav) are captured concurrently, each stamped with both an episode-relative timestamp and an absolute Unix timestamp at its native rate.
- 03
Timestamp normalization
All streams carry float64 Unix and episode-relative timestamps for cross-stream alignment. Native measurement rates are preserved rather than resampled, with a sync-tolerance budget recorded in the format.
- 04
Quality and success labeling
Each episode gets a recording-quality tag and an active-stream count, and tasks carry a boolean success plus failure reason. Accepted episodes are organized by outcome, and admin review gates customer visibility.
- 05
Licensing and watermark
On delivery each episode is stamped with a license block, a multi-layer watermark, a cryptographic signature, and per-file checksums.
- 06
ORTF packaging
Episodes are packaged in the Open Robot Training Format: a manifest, an episode index, chunked step records, videos, and optional annotations and robot description, convertible to LeRobot and RLDS / Open-X-Embodiment.
Request a sample
We send real records in the format you would receive, with the field list and where each record came from. Access for evaluation is limited and covered by a mutual NDA.
ProductRobot Teleoperation Demonstrations