A standardized, open file format for robot training data. Designed for interoperability across research labs, robotics companies, and foundation model training pipelines.
Share datasets between labs, companies, and training pipelines without conversion headaches.
Single ZIP archive contains everything: video, sensors, metadata, and checksums for verification.
Designed for direct ingestion into training pipelines. Compatible with HuggingFace, LeRobot, and RLDS.
episode_001.ortf.zip
├── manifest.json # Episode metadata & file checksums
├── video/
│ ├── rgb_main.mp4 # Primary RGB camera (H.264, 30fps)
│ ├── rgb_wrist.mp4 # Wrist-mounted camera (optional)
│ └── rgb_overhead.mp4 # Overhead view (optional)
├── depth/
│ ├── depth_main.npz # Depth frames (16-bit, aligned to RGB)
│ └── depth_timestamps.json
├── audio/
│ └── audio.wav # Microphone capture (16kHz mono)
├── proprioception/
│ ├── joints.jsonl # Joint positions, velocities, torques
│ └── state.jsonl # End-effector pose, gripper state
├── sensors/
│ ├── imu.jsonl # IMU accelerometer & gyroscope
│ ├── tactile.jsonl # Tactile sensor array (if available)
│ └── lidar.jsonl # LiDAR scans (if available)
├── tracking/
│ ├── hand_data.json # Hand tracking landmarks (synthetic)
│ └── pose_data.json # Full-body pose estimation
└── annotations/
├── task.json # Task description & success label
└── scene_objects.json # Bounding boxes & object labels{
"ortf_version": "1.0",
"episode_id": "episode_001",
"created_at": "2024-12-19T10:30:00Z",
"data_type": "episode", // episode | synthetic_pov | synthetic_third_person | simulation
"task": {
"instruction": "Pick up the red cube and place it in the basket",
"description": "Object manipulation with precision grasping",
"success": true,
"failure_reason": null
},
"robot": {
"id": "robot_001",
"type": "manipulator",
"manufacturer": "Booster Robotics",
"model": "T1",
"dof": 7
},
"timing": {
"duration_seconds": 12.5,
"start_unix": 1703000000.000,
"end_unix": 1703000012.500
},
"sensors": {
"rgb_main": {
"file": "video/rgb_main.mp4",
"codec": "h264",
"fps": 30,
"resolution": [1920, 1080],
"frame_count": 375
},
"depth_main": {
"file": "depth/depth_main.npz",
"format": "uint16",
"unit": "millimeters",
"frame_count": 375
},
"joints": {
"file": "proprioception/joints.jsonl",
"rate_hz": 100,
"sample_count": 1250,
"fields": ["position", "velocity", "torque"]
}
},
"files": {
"checksums": {
"video/rgb_main.mp4": "sha256:a1b2c3d4...",
"depth/depth_main.npz": "sha256:e5f6g7h8...",
"proprioception/joints.jsonl": "sha256:i9j0k1l2..."
},
"total_size_bytes": 12400000
}
}Primary visual stream from robot cameras. Multiple views supported.
Per-pixel depth aligned to RGB frames. Millimeter precision.
Joint states, end-effector pose, and gripper data at high frequency.
Microphone capture for contact sounds and environmental audio.
Inertial measurement from accelerometer and gyroscope.
Touch sensor arrays from gripper or robot skin.
3D hand landmark positions for synthetic demonstrations.
Full-body pose landmarks for third-person synthetic data.
// Each line is a JSON object:
{"ts": 1703000000.000, "pos": [0.12, -0.34, 0.78, 0.45, -0.23, 0.91, 0.02], "vel": [...], "torque": [...]}
{"ts": 1703000000.010, "pos": [0.12, -0.34, 0.78, 0.45, -0.23, 0.91, 0.02], "vel": [...], "torque": [...]}
{"ts": 1703000000.020, "pos": [0.13, -0.33, 0.77, 0.46, -0.22, 0.90, 0.03], "vel": [...], "torque": [...]}
...
// Fields:
// ts: Unix timestamp with nanosecond precision (float)
// pos: Joint positions in radians [j1, j2, j3, j4, j5, j6, j7]
// vel: Joint velocities in rad/s
// torque: Joint torques in Nm
// gripper: Gripper position 0.0 (closed) to 1.0 (open)Each line is a JSON object with nanosecond-precision timestamps. Positions in radians, velocities in rad/s, torques in Nm.
Every ORTF archive includes SHA-256 checksums for all files in the manifest. This ensures data integrity during transfer and storage.
"files": {
"checksums": {
"video/rgb_main.mp4": "sha256:a1b2c3d4e5f6789...",
"depth/depth_main.npz": "sha256:9876fedcba4321...",
"proprioception/joints.jsonl": "sha256:abcdef123456..."
},
"manifest_version": "1.0",
"total_size_bytes": 12400000
}Human-operated robot recordings with full sensor capture
AI-generated first-person demonstrations with hand tracking
AI-generated third-person view with pose estimation
Physics engine recordings (MuJoCo, Isaac, etc.)
Single episode of a pick-and-place task with RGB video, depth, and 7-DOF joint states.
Synthetic POV video with MediaPipe hand tracking and 21-landmark 3D coordinates.
Use our Python validator to check your ORTF archives for compliance:
pip install ortf-validator
ortf-validate my_episode.ortf.zipfrom ortf import load_ortf
ds = load_ortf("episode.ortf.zip")
ds.push_to_hub("my-robot-data")Direct conversion to HuggingFace Dataset format
from ortf.lerobot import convert
convert("episode.ortf.zip",
output_dir="lerobot_data/")Export to LeRobot dataset structure
from ortf.rlds import to_tfds
ds = to_tfds("episode.ortf.zip")
# TensorFlow Dataset readyConvert to RLDS/TensorFlow Datasets
We thank the following researchers and engineers for their feedback on this specification:
Interested in reviewing this specification?
Contact Us →Join the growing community of robotics researchers and companies using the Open Robot Training Format for interoperable training data.