Open standard

Open Robot Training Format

A standardized, open file format for robot training data. Designed for interoperability across research labs, robotics companies, and foundation model training pipelines.

View the specification View examples

Version: 1.0
Container: ZIP
Modalities: 7+
Metadata: JSON schema
Draft: ORTF v0.2 specification
Product: Robot Teleoperation Demonstrations

Why a standard format

Interoperability: Share data between labs, companies, and training pipelines without conversion headaches.
Self-contained: Single ZIP archive contains everything: video, sensors, metadata, and checksums for verification.
ML-ready: Designed for direct ingestion into training pipelines. Compatible with HuggingFace, LeRobot, and RLDS.

File format structure

ORTF archive structure

episode_001.ortf.zip
├── manifest.json           # Episode metadata & file checksums
├── video/
│   ├── rgb_main.mp4        # Primary RGB camera (H.264, 30fps)
│   ├── rgb_wrist.mp4       # Wrist-mounted camera (optional)
│   └── rgb_overhead.mp4    # Overhead view (optional)
├── depth/
│   ├── depth_main.npz      # Depth frames (16-bit, aligned to RGB)
│   └── depth_timestamps.json
├── audio/
│   └── audio.wav           # Microphone capture (16kHz mono)
├── proprioception/
│   ├── joints.jsonl        # Joint positions, velocities, torques
│   └── state.jsonl         # End-effector pose, gripper state
├── sensors/
│   ├── imu.jsonl           # IMU accelerometer & gyroscope
│   ├── tactile.jsonl       # Tactile sensor array (if available)
│   └── lidar.jsonl         # LiDAR scans (if available)
├── tracking/
│   ├── hand_data.json      # Hand tracking landmarks (synthetic)
│   └── pose_data.json      # Full-body pose estimation
└── annotations/
    ├── task.json           # Task description & success label
    └── scene_objects.json  # Bounding boxes & object labels

manifest.json schema

Manifest schema

{
  "ortf_version": "1.0",
  "episode_id": "episode_001",
  "created_at": "2024-12-19T10:30:00Z",

  "data_type": "episode",  // episode | synthetic_pov | synthetic_third_person | simulation

  "task": {
    "instruction": "Pick up the red cube and place it in the basket",
    "description": "Object manipulation with precision grasping",
    "success": true,
    "failure_reason": null
  },

  "robot": {
    "id": "robot_001",
    "type": "manipulator",
    "manufacturer": "Booster Robotics",
    "model": "T1",
    "dof": 7
  },

  "timing": {
    "duration_seconds": 12.5,
    "start_unix": 1703000000.000,
    "end_unix": 1703000012.500
  },

  "sensors": {
    "rgb_main": {
      "file": "video/rgb_main.mp4",
      "codec": "h264",
      "fps": 30,
      "resolution": [1920, 1080],
      "frame_count": 375
    },
    "depth_main": {
      "file": "depth/depth_main.npz",
      "format": "uint16",
      "unit": "millimeters",
      "frame_count": 375
    },
    "joints": {
      "file": "proprioception/joints.jsonl",
      "rate_hz": 100,
      "sample_count": 1250,
      "fields": ["position", "velocity", "torque"]
    }
  },

  "files": {
    "checksums": {
      "video/rgb_main.mp4": "sha256:a1b2c3d4...",
      "depth/depth_main.npz": "sha256:e5f6g7h8...",
      "proprioception/joints.jsonl": "sha256:i9j0k1l2..."
    },
    "total_size_bytes": 12400000
  }
}

Supported modalities

Modality	Format	Description	Specification
RGB Video	MP4 (H.264)	Primary visual stream from robot cameras. Multiple views supported.	1080p @ 30fps, wrist cam, overhead, third-person
Depth	NPZ (uint16)	Per-pixel depth aligned to RGB frames. Millimeter precision.	16-bit, aligned, range: 0.3-10m
Proprioception	JSONL	Joint states, end-effector pose, and gripper data at high frequency.	100Hz+, position, velocity, torque
Audio	WAV	Microphone capture for contact sounds and environmental audio.	16kHz, mono, PCM
IMU	JSONL	Inertial measurement from accelerometer and gyroscope.	1000Hz, 3-axis accel, 3-axis gyro
Tactile	JSONL	Touch sensor arrays from gripper or robot skin.	100Hz, pressure map, contact detection
Hand Tracking	JSON	3D hand landmark positions for synthetic demonstrations.	21 landmarks, per-hand, MediaPipe
Pose Estimation	JSON	Full-body pose landmarks for third-person synthetic data.	33 landmarks, world coords, meters

Proprioception format (JSONL)

joints.jsonl

// Each line is a JSON object:
{"ts": 1703000000.000, "pos": [0.12, -0.34, 0.78, 0.45, -0.23, 0.91, 0.02], "vel": [...], "torque": [...]}
{"ts": 1703000000.010, "pos": [0.12, -0.34, 0.78, 0.45, -0.23, 0.91, 0.02], "vel": [...], "torque": [...]}
{"ts": 1703000000.020, "pos": [0.13, -0.33, 0.77, 0.46, -0.22, 0.90, 0.03], "vel": [...], "torque": [...]}
...

// Fields:
// ts: Unix timestamp with nanosecond precision (float)
// pos: Joint positions in radians [j1, j2, j3, j4, j5, j6, j7]
// vel: Joint velocities in rad/s
// torque: Joint torques in Nm
// gripper: Gripper position 0.0 (closed) to 1.0 (open)

Each line is a JSON object with nanosecond-precision timestamps. Positions in radians, velocities in rad/s, torques in Nm.

Integrity verification

Every ORTF archive includes SHA-256 checksums for all files in the manifest. This ensures data integrity during transfer and storage.

Checksums in manifest.json

"files": {
  "checksums": {
    "video/rgb_main.mp4": "sha256:a1b2c3d4e5f6789...",
    "depth/depth_main.npz": "sha256:9876fedcba4321...",
    "proprioception/joints.jsonl": "sha256:abcdef123456..."
  },
  "manifest_version": "1.0",
  "total_size_bytes": 12400000
}

Supported episode types

data_type	Episode type	Description
`episode`	Teleoperation	Human-operated robot recordings with full sensor capture
`synthetic_pov`	Synthetic POV	AI-generated first-person demonstrations with hand tracking
`synthetic_third_person`	Third Person	AI-generated third-person view with pose estimation
`simulation`	Simulation	Physics engine recordings (MuJoCo, Isaac, etc.)

Sample data

Pick & Place Demo

Single episode of a pick-and-place task with RGB video, depth, and 7-DOF joint states.

episode, rgb, depth, joints

View the teleop sample

Hand Tracking POV

Synthetic POV video with MediaPipe hand tracking and 21-landmark 3D coordinates.

synthetic_pov, hand_tracking, video

Browse synthetic POV

Validate your ORTF files

To validate files against the specification, email ortf@gerra.com.

Framework integration

HuggingFace Datasets

from ortf import load_ortf
ds = load_ortf("episode.ortf.zip")
ds.push_to_hub("my-robot-data")

Direct conversion to HuggingFace Dataset format

LeRobot

from ortf.lerobot import convert
convert("episode.ortf.zip",
        output_dir="lerobot_data/")

Export to LeRobot dataset structure

RLDS / Open X

from ortf.rlds import to_tfds
ds = to_tfds("episode.ortf.zip")
# TensorFlow Dataset ready

Convert to RLDS/TensorFlow Datasets

The reference implementation is shared on request from ortf@gerra.com. It is not published on PyPI.

Format reviewers

Reviewers will be listed here when the review is done. To review the specification, email ortf@gerra.com.

Source and samples

Browse sample data