First-Person Human Motion
First-person video of people doing tasks, paired with full-body motion capture.
Sample
3 recordsThird-person body track, ball toss (33 landmarks)
Representative of the real pose file. pose_data holds 3 coordinates per landmark in world meters.
{
"metadata": {
"num_frames": 39,
"sample_rate": 5,
"num_landmarks": 33,
"coords": "world",
"units": "meters"
},
"connections": [[0, 1], [1, 2], [2, 3], [3, 7], [0, 4]],
"pose_data": [[[-0.0349, -0.6541, -0.2357], [-0.0195, -0.69, -0.2276]]]
}Clip manifest, POV egocentric
Representative. First-person type with a MediaPipe 33-landmark body pose.
{
"episode": {
"id": "pov_20251112_233840",
"type": "pov",
"prompt": "Configuring a home router",
"duration_sec": 8
},
"video": {
"raw": "raw/video.mp4",
"codec": "h264",
"fps": 24,
"resolution": [1280, 720],
"frame_count": 192
},
"pose_3d": {
"method": "mediapipe",
"keypoints_uri": "raw/body_keypoints_3d.npz",
"num_landmarks": 33,
"num_frames": 192,
"coords": "world",
"units": "meters"
}
}First-person hand keypoints (21 landmarks per hand)
Representative of the real packed-array shape; 21 landmarks per hand x 3 coordinates, with per-frame visibility.
left_hand: float64 [1892, 21, 3] # XYZ world meters right_hand: float64 [1892, 21, 3] left_visibility: float64 [1892] # per-frame confidence right_visibility: float64 [1892] landmark_names: [21] (WRIST, THUMB_TIP, INDEX_TIP, ...) connections: int [23, 2] # hand skeleton edges fps: int
Specifications
| Formats | H.264, BVH, FBX, JSONL |
|---|---|
| Delivery | S3, REST API, CDN, Public preview URLs |
| Cadence | Continuous collection, batch delivery |
| Coverage | 2023 to 2026 |
| Vision | 1920x1080 @ 30fps |
| Motion | Rokoko full-body 3D skeletal |
| Viewpoints | First-person + external, synchronized |
| Annotation | 3D pose landmarks |
| Sync | <1ms across streams |
| Access | The public sample tier is open via preview URLs; the full product is behind org-scoped API keys with tenant isolation. |
What ships
First-Person POV
Egocentric human video of real tasks - the visual prior for embodied perception and affordance learning.
Full-Body Motion Capture
Rokoko-suit 3D skeletal tracking with joint-level body landmarks, time-synced to video.
Multi-View Sync
First-person and external viewpoints aligned to the same timeline for cross-view learning.
Fields
14 fields
| Field | Description |
|---|---|
| episode.idstring | Clip identifier.Example syn_third_20251119_051421 |
| episode.typestring | Egocentric (POV) vs external (third-person).Example pov |
| episode.promptstring · nullable | Task or activity label.Example Configuring a home router |
| episode.duration_secfloat · s | Clip duration.Example 8.0 |
| video.codecstring | Video codec.Example h264 |
| video.fpsint · fps | Frame rate.Example 24 |
| video.resolution[w,h] · px | Frame size.Example [1280, 720] |
| video.frame_countint · frames | Total frames.Example 192 |
| pose_3d.methodstring | Pose estimator.Example mediapipe |
| pose_3d.num_landmarksint | Body landmark count (33 body, 21 per hand).Example 33 |
| pose_3d.coordsstring | Coordinate frame.Example world |
| pose_3d.unitsstring | Spatial units.Example meters |
| pose_datafloat[F][L][3] · m | Per-frame, per-landmark XYZ world coordinates.Example [[[-0.0349,-0.6541,-0.2357], ...]] |
| connectionsint[][2] | Skeleton edge list (landmark index pairs).Example [[0,1],[1,2],[2,3]] |
Load it
The first commands after an approved delivery.
# Load a human-POV clip and its 3D skeleton # 1. Resolve a sample's urls (video, pose_data, keypoints_3d, preview_gif) # 2. Download video.mp4 and load body_keypoints_3d.npz with NumPy # 3. Read pose_data [F, 33, 3] + connections to render the skeleton
Quality checks
3 checksPose tracking confidence
Whether extracted skeletal keypoints are reliable per frame.
Per-landmark visibility scores are stored alongside keypoints and tracked as a QA metric.
Methodology-stage. Per-frame visibility is recorded; no published aggregate accuracy.
Skeleton structural validity
Whether keypoint arrays conform to the expected landmark count and connection graph.
Validate the frame x landmark x 3 array shape against the declared landmark count (33 body, 21 hand) and the connection edge list.
Methodology-stage. Shape and consistency check.
AI annotation QA
Whether the auto-generated task, object, and narration labels are correct.
Generated metadata is reviewed in admin QA before customer visibility.
Methodology-stage. Human review gate; no scored metric.
Ground truth
| What is correct | The human actual demonstrated motion is the ground truth, and the MediaPipe 3D skeleton plus 21-landmark-per-hand track is its structured representation. Task and object labels are graded against human-reviewed annotation. |
|---|---|
| How it is established | Per-frame visibility scores flag low-confidence keypoints, admin human QA reviews enrichment outputs, and structural validity is checked against the declared landmark and connection schema. |
| Agreement | No inter-rater or keypoint-accuracy agreement statistic is published. |
How it is built
- 01
Egocentric and external capture
Human demonstrations are captured as first-person (POV) and external (third-person) video of real tasks in H.264.
- 02
Pose estimation
MediaPipe extracts 3D skeletal keypoints: 33 body landmarks for the full body and 21 landmarks per hand for fine manipulation, in world coordinates in meters.
- 03
Skeleton structuring
Keypoints are stored as per-frame XYZ arrays with an explicit connection edge list defining the skeleton graph, plus per-landmark visibility.
- 04
AI enrichment
A vision-language model auto-generates the task name, description, objects present, and actions, with timestamped narration for language-conditioned policies.
- 05
Preview and packaging
Each clip is rendered to a preview GIF with tracking overlay and an annotated pose frame, bundled with the raw video, pose JSON, packed keypoints, and a manifest.
- 06
Public sample surfacing
Real human-POV clips are kept as their own line, served from storage object URLs, separate from any synthetic or generated POV.
Request a sample
We send real records in the format you would receive, with the field list and where each record came from. Access for evaluation is limited and covered by a mutual NDA.
ProductFirst-Person Human Motion