Chapter 2: What Do We Pretrain On? — Robot Data, Cross-Embodiment Learning, and Action Representations

2.1 Learning Objectives and Experimental Question

The useful material of a robot foundation model is not a count of videos or trajectories. It is experience whose semantics survive the pipeline. By the end of this chapter, the reader should be able to distinguish what robot episodes, human video, language annotations, simulation, and synthetic data do and do not teach; choose an action representation that matches the task and controller; and audit where information is lost in cross-embodiment learning.

After reading this chapter... you will be able to compare missing semantics across data sources, quarantine episodes with incompatible time/frame/control meanings, choose an action representation for each assembly phase, and decide promotion with a mixture receipt and embodiment-transform audit.

The through-task remains tabletop assembly with one arm and a parallel gripper: grasp a part, transport it without collision, stop at a pre-contact pose, and insert it into a fixture under bounded force. The operative question is not “Which dataset is largest?” It is: after mixing these sources, do the coordinate, timing, contact, and failure semantics needed for insertion remain recoverable, and can every model output return to the inherited execution and safety contract?

Inherited Inputs and New Outputs

• S11 inheritance: verify the actual robot, gripper, and camera identities; calibration versions; frames and clocks; control rate; collision and force limits; watchdog; protective stop; E-stop; and responsible human.
• S12 inheritance: freeze the versioned task and skill APIs, observation/action schemas, proposal → projected → sent → accepted/executed log, feasibility projection, contact controller, independent evaluator, and rollback tuple.
• S13 addition: produce a data/action-representation decision table, a mixture receipt, and an embodiment-transform audit from each source embodiment to the default cell.

If an inherited field is not actually available, the model must not fill it in. Record missing, not measured, not exercised, or incompatible, and close the corresponding promotion gate.

2.2 Responsibility Model: A Common Representation Is Not a Common Controller

The data pipeline has five accountable layers.

This is more than documentation hygiene. A normalization layer may express six-degree-of-freedom end-effector deltas as a “common action,” but the joint velocities, singularity margin, gripper delay, and contact-control mode that realize the delta remain robot-specific. A shared representation is a learning interface, not shared physics or a shared safety proof.

2.3 What Enters the Mixture: Comparing Sources on Common Axes

Robot-learning sources are not substitutes. They are measurements with different missing fields. Early large-scale self-supervised grasping supplied real images and action outcomes but covered a narrow gripper and task distribution Pinto & Gupta, 2016. Multi-robot video increased embodiment diversity but still required platform-specific camera and action processing Dasari et al., 2020. Visual representations pretrained on human video can improve perceptual sample efficiency with few robot demonstrations, yet provide neither robot actions nor forces and contact outcomes Nair et al., 2022.

Domain randomization can make a real scene look like one training variation, but visual randomization alone does not transfer dynamics or contact Tobin et al., 2017. UMI offers a practical route from portable human demonstrations to robot policies, while explicitly identifying human actions that are valid yet infeasible on target hardware as an open problem Chi et al., 2024. “Includes human data” therefore means that useful observation and intent signals have been added—not that the robot action space has been filled.

How to Read Scale Claims

Open X-Embodiment reports 22 robots, 21 institutions, 527 skills, and 160,266 tasks, demonstrating breadth across real-hardware sources; it does not establish that their controllers, contact conditions, failures, or evaluation protocols form one denominator Open X-Embodiment Collaboration, 2024, §§S3–S6 and Tables S4.T1/S5.T2. Preserve these counts with the paper's definitions and version, and never merge them into the S13 tabletop-assembly success denominator.

DROID reports 76,000 trajectories and 350 hours across 564 scenes and 86 tasks, but these source-specific counts do not establish equivalent coverage of failures, interventions, contact events, or train–evaluation overlap Khazatsky et al., 2024, arXiv:2403.12945v2. Count successful episodes per hour separately from learner-visited failure states.

The counterexample to naive scaling is straightforward. Adding only optimal demonstrations improves coverage of normal paths while scarcely observing a rising-force state after misalignment or a state at which a human stopped execution. DAgger addresses covariate shift by requesting expert labels on states visited by the learner, but a safety-critical robot cannot deliberately visit every dangerous state merely to obtain a label Ross et al., 2011. Combine shadow execution, existing failure logs, bounded intervention collection, and simulated counterexamples instead.

2.4 Align Schema and Time Before Learning

A minimally auditable episode contains:

episode_id, source_id, task_version, embodiment_id, calibration_id
observation: value, unit, frame_id, capture_time, arrival_time, validity
action: proposal, projected, sent, accepted, executed, control_mode
event: contact, intervention, protective_stop, e_stop, timeout, reset
outcome: goal_state, failure_class, evaluator_version, human_override
lineage: raw_hash, transform_version, split_id, license_snapshot

Collapsing event, arrival, and execution time into one timestamp makes it impossible to detect a new action paired with an old image. Record observation age a_t=t_{execute}-t_{capture}, proposal latency, queue delay, and controller-acceptance delay separately. Two sources are not equivalent at “10 Hz” if one has larger jitter, missing samples, or different command-hold behavior.

Cross-robot learning requires explicit mappings for action coordinates, units, frequency, controller mode, and gripper semantics; source-wise normalization in Open X-Embodiment, adaptation to new observation/action spaces in Octo, and the human–robot interface in UMI do not establish a lossless universal map Open X-Embodiment Collaboration, 2024; Octo Model Team et al., 2024; Chi et al., 2024.

For example, gripper=1 may mean close in one source, open in another, and a target width of 1 m in a third. End-effector poses change meaning depending on whether they are absolute or delta commands and whether the frame is world, base, camera, or tool. Never infer missing semantics from data statistics. Quarantine the source as incompatible.

Time and Schema Gates

2.5 Choosing an Action Representation on Common Axes

Greater compression may help learning and long context but erase the fine correction and deadline information needed during insertion. Per-step continuous output preserves detail but can be vulnerable to inference delay, noise, and mode averaging. Implicit Behavioral Cloning represents multiple action modes with an energy model, but introduces sampling cost and does not guarantee out-of-support calibration Florence et al., 2021. The flow-based \pi_0 provides another route to high-frequency continuous action distributions, yet its authors report that broad pretraining alone does not reliably supply recovery from mistakes Black et al., 2024.

For S13's design audit, continuous actions, action chunks, action tokens, latent actions, and skill calls are compared on common axes of detail retention, compression, multimodality, and inference/execution latency. Diffusion Policy supports multimodal continuous action sequences, π0 predicts continuous action chunks with flow matching, and FAST studies compression-based action tokenization; these sources do not provide a matched default-cell comparison of all five representations Chi et al., 2023; Black et al., 2024; Pertsch et al., 2025. Select among them with the system axes below, not with cross-paper success rankings.

FAST trains a general tokenizer on one-second action chunks from multiple robots, but “can be applied” does not mean that target-cell contact precision and deadlines are preserved Pertsch et al., 2025. A mean position reconstruction metric can conceal rare gripper transitions and rotational errors immediately before insertion. Measure error by task phase, tails, contact intervals, and controller saturation.

Default Selection for Assembly

Use a hierarchy. The language-conditioned model proposes a versioned skill call or a short end-effector trajectory. Permit action chunks during free-space transport, but discard the active chunk at the pre-contact stop and infer again from the latest observation. On insertion, use short continuous references verified on the target cell and the S12 impedance/force controller. Tokens and latent actions may serve as training representations, but decoding never bypasses feasibility projection.

2.6 Dataset Mixtures and Sampling Weights

If a source d receives mixture weight simply in proportion to episode count, the robot that was recorded longest or sampled fastest can dominate the objective. Record the effective sampling probability at least as explicitly as:

$$

p(d,k,y) \propto w_d\,w_k\,w_y\,q^{-1}_{dup}\,q_{valid},

$$

where k is the assembly phase, y is the success/failure/intervention class, q_{dup} is the near-duplicate cluster size, and q_{valid} represents sensor, timing, and license validity. This expression is not a universally optimal recipe. It is a receipt that makes repeated exposure reproducible.

Design the mixture in this order:

1. Build per-source histograms over task, embodiment, scene, instruction, and outcome axes.
2. Cluster exact and near duplicates, including adjacent capture segments, before splitting.
3. Sample successful demonstrations separately from failure, intervention, and recovery strata.
4. Report contribution in episodes, time, and contact events—not only camera frames.
5. Monitor source-wise loss, gradient contribution, minority-source performance, and default-cell performance.
6. Version every mixture change with a new dataset and checkpoint while preserving the previous recipe.

Bridge Data shows that broader environments and tasks can help downstream imitation, but its common embodiment limits evidence for morphology transfer, and diversity does not prove absence of visual or task overlap Ebert et al., 2021. Octo likewise identifies the limitations of an optimal-demonstration-heavy mixture, including wrist-camera support, language conditioning, and the absence of suboptimal data Octo Model Team et al., 2024. Blindly oversampling scarce failures can amplify sensor faults or incorrect interventions, so failure taxonomy and quality review precede weighting.

2.7 Embodiment-Transform Audit: What Transfers and What Remains Local

Object appearance, linguistic goal relations, some visual features, coarse task phases, and outcome states are relatively shareable. Joint count and limits, reachable sets, tool-center points, gripper mechanisms, tactile/force observations, control modes, command rates, and contact stability are not directly shareable.

Cross-embodiment evidence must state what changed in kinematics, sensing, end effector, action schema, and control; humanoid and bimanual results are advanced branches, not direct evidence for the default single-arm parallel-gripper cell Open X-Embodiment Collaboration, 2024; Octo Model Team et al., 2024; NVIDIA et al., 2025.

Tabletop-Assembly Embodiment-Transform Audit

An embodiment transform is not one matrix; it is the versioned bundle represented by this table. A model that accepts a new camera or a fine-tuned action head has not transferred gripper-force semantics or protective-stop responsibility. Positive transfer in Open X-Embodiment and Octo justifies attempting this audit, not omitting it.

2.8 Provenance, Rights, and Contamination Ledger

Data lineage is a directed graph from raw → cleaned → synchronized → transformed → mixed → split → checkpoint → evaluation, not a file path. Each edge records code version, arguments, input/output hashes, and lossy fields. Split at the cluster level when ostensibly different files depict the same raw video, adjacent segments from one capture, or paraphrases of the same instruction.

S13 therefore audits code, data, weights, images/video, and annotations as separate asset classes. The Open X-Embodiment repository licenses software under Apache 2.0 and other repository materials under CC BY 4.0, while the Octo and DROID policy-learning repositories label their code MIT; those repository-level grants do not by themselves license every constituent dataset, weight, media item, or annotation. If a complete training manifest is unavailable, overlap remains possible rather than proven absent.

A license snapshot records check date, authoritative URL, asset class, research/commercial use, modification, redistribution, attribution, consent, and privacy separately. Mark anything requiring legal judgment review_required; do not infer permission. The contamination audit includes exact hashes, perceptual hashes, temporal adjacency, scene/object/instruction identifiers, and any known Internet-pretraining inclusion. When the manifest is unavailable, the evaluation card says possible overlap; manifest unavailable.

2.9 Minimal Reproducible Workflow and Promotion Gates

This workflow is designed to fail the data contract before training an expensive model.

1. Freeze: version the assembly task, initial-state distribution, skill APIs, target cell, and evaluator.
2. Inventory: record source assets, license, consent, raw hashes, device IDs, and calibration IDs.
3. Validate schemas: quarantine episodes missing units, frames, three timelines, control mode, or gripper semantics.
4. Replay alignment: regenerate normalized data from raw records and inspect image–state–action lag numerically and visually.
5. Deduplicate and split: form near-duplicate clusters before combining sources; split by cluster.
6. Issue a mixture receipt: record weights and effective sample counts by source, phase, and outcome.
7. Round-trip the representation: encode and decode actions; measure position, rotation, gripper, phase-wise contact error, and deadline misses.
8. Test embodiment transforms: replay through target IK, constraint projection, and controller acceptance while preserving rejection reasons.
9. Promote evidence: proceed through offline→replay→simulation→shadow→human-approved bounded hardware trials.
10. Exercise rollback: restore the complete tuple of dataset, transform, mixture, model/tokenizer, controller, and evaluator versions.

Even in a bounded trial, the model output remains a proposal. Feasibility projection, collision/force gates, the real-time controller, independent evaluator, watchdog, protective stop, E-stop, and human authority are not inherited from data or model weights.

2.10 Diagnosing Failures from Symptom to Boundary

Do not end diagnosis with “the model is bad.” With the same checkpoint, compare raw-frame replay, normalized replay, target inverse mapping, and feasibility projection in order. Find the first boundary that diverges before any physical test.

2.11 Evidence, Counterevidence, and Remaining Limits

Large-scale real-robot collection broadened visual and action experience Levine et al., 2016, while multi-task and offline–online mixtures showed that experience can be reused across related skills Yu et al., 2021; Lu et al., 2022. Yet narrow grasp distributions and substantial reset or human-exposure costs disappear easily in a scale table. The controlled robomimic study further shows that demonstration quality, observation modalities, and training choices materially affect offline imitation outcomes; an offline success rate alone does not expose reset and intervention costs Mandlekar et al., 2021.

The inspected evidence does not show that:

• any one dataset or representation validates the complete S13 tabletop-assembly stack;
• success rates from different robots, tasks, data volumes, adaptation protocols, resets, and evaluators form a leaderboard;
• action normalization losslessly preserves force, tactile, compliance, contact, and latency semantics;
• an unknown training manifest constitutes a clean evaluation;
• broad human, video, or simulated data grants feasibility or safety authority on target hardware; or
• tail observation age, queue/stale behavior, reconnect semantics, remote outages, and exact rollback are reported in most source papers.

The conclusion is not to avoid large mixtures. It is to learn broadly from shareable representations while reinstating lost physical semantics through embodiment-specific adapters, validation, and the inherited execution contract.

2.12 Metrics and Artifacts to Record

Metrics

• Episodes, hours, contact events, successes, failures, interventions, and recoveries by source, task phase, and embodiment
• Valid-observation rate, missingness, jitter, median/p95/p99 observation age, and stale-command rejection rate
• Mean, p95, and maximum frame/unit round-trip error and action reconstruction error
• Phase-wise error at gripper transitions, pre-contact stop, and insertion
• Duplicate-cluster count, exact and near overlap across splits, and unavailable-manifest ratio
• Source sampling probability, effective sample count, loss/gradient contribution, and minority-source performance
• Feasibility rejection, controller rejection/saturation, collision/force events, protective stops, and human interventions
• Success and failure denominator per embodiment, recovery success, inference/queue/execution latency, and cost

Required Artifacts

• dataset_manifest: sources, versions, hashes, assets, licenses, consent
• schema_registry: observation, action, units, frames, time, control, gripper semantics
• transform_receipt: raw→common→target transforms, losses, and tests
• split_overlap_report: duplicate clusters and train/validation/evaluation overlap
• mixture_receipt: weights, effective counts, success/failure/intervention composition
• action_representation_card: representation, tokenizer, normalization, round-trip error, latency
• embodiment_transform_audit: the audit table above plus target-cell tests
• promotion_and_rollback_receipt: evidence stage, approver, exact rollback tuple

2.13 Bounded Codex Implementation Prompt

Goal:
Perform a read-only audit of the robot-motion-101-part-3 tabletop-assembly data.
Draft dataset_manifest, schema_registry, mixture_receipt,
action_representation_card, and embodiment_transform_audit.

Context:
- The target is the default single-arm + parallel-gripper cell.
- Inputs include the actual S11 cell/calibration/time/safety versions and the
  S12 skill-API/execution-log/evaluator contract.
- Model output is a bounded proposal, never command authority.

Constraints:
- Do not modify raw data, splits, models, or robot configuration; send no commands.
- Do not infer units, frames, time semantics, control mode, or gripper meaning.
- Record unknowns as missing/not measured/not exercised/incompatible.
- Do not use a paper, code, data, weights, or media license as a substitute for another.
- Report exact/near duplicates by cluster; never infer that evaluation is clean.

Done when:
1) Every source has hashes, versions, and rights status.
2) Three timelines and proposal/projected/sent/accepted/executed are separated.
3) Action round-trip error is reported for position, rotation, gripper, and contact phases.
4) Every embodiment-audit row is pass/fail/missing.
5) Mixture weights and effective sample counts are reproducible.
6) Closed gates, failure reasons, and the exact rollback tuple are summarized.

Safety:
Do not change collision/force gates, watchdog, protective stop, E-stop, or human approver.
Reject hardware promotion if any required field remains unverified.

2.14 Terry's Related Reading

These are exact reader cross-links for further context; they are not evidence for this chapter's claims.

• \pi_0: VLA and Flow-Based Action Modeling
• DexForce: Force-Informed Actions
• RoboPaint
• Gen-0 Embodied Foundation Model
• Gen-1 and Scaling Mastery
• ACT-1 Robot Foundation Model
• UMI: Universal Manipulation Interface
• UMI-FT: Compliant Manipulation
• Octo Generalist Policy

2.15 Bridge to the Next Chapter

This chapter established which data and action encodings are eligible to enter a model. The next chapter holds these input contracts fixed while comparing how visual and language encoders, multimodal fusion, and policy heads turn the material into bounded proposals. Regardless of model family, this chapter's lineage, embodiment transforms, time alignment, feasibility projection, and independent safety authority remain unchanged.

Supplementary evidence map

The following verified primary-source groups come from the independent S13 ledger. They document search coverage without changing existing claims or denominators.

• data and imitation-learning lineages: BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning, PaLM-E: An Embodied Multimodal Language Model, AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale, Behavior Transformers: Cloning k Modes with One Stone — source-specific tasks, data, and evidence stages remain separate.
• uncertainty and confidence: Masked Visual Pre-training for Motor Control, ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control, VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Models — source-specific tasks, data, and evidence stages remain separate.
• world models and action generation: Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization, World Action Models are Zero-shot Policies, FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation — source-specific tasks, data, and evidence stages remain separate.

References

1. Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009). A Survey of Robot Learning from Demonstration. Robotics and Autonomous Systems.
2. Ross, S., Gordon, G., & Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS. arXiv:1011.0686.
3. Pinto, L., & Gupta, A. (2016). Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours. IEEE ICRA. arXiv:1509.06825.
4. Levine, S., Pastor, P., Krizhevsky, A., & Quillen, D. (2016). Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. ISER. arXiv:1603.02199.
5. Finn, C., Goodfellow, I., & Levine, S. (2016). Unsupervised Learning for Physical Interaction through Video Prediction. NeurIPS. arXiv:1605.07157.
6. Finn, C., Yu, T., Zhang, T., Abbeel, P., & Levine, S. (2017). One-Shot Visual Imitation Learning via Meta-Learning. CoRL. arXiv:1703.07326.
7. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IEEE IROS. arXiv:1703.06907.
8. Dasari, S., et al. (2020). RoboNet: Large-Scale Multi-Robot Learning. CoRL. arXiv:1910.11215.
9. Ebert, F., Nair, S., Lee, A., Levine, S., & Finn, C. (2021). Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets. CoRL. arXiv:2109.13396.
10. Florence, P., et al. (2021). Implicit Behavioral Cloning. CoRL. arXiv:2109.00137.
11. Mandlekar, A., et al. (2021). What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. CoRL. arXiv:2108.03298.
12. Yu, T., et al. (2021). MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. arXiv:2104.08212.
13. Lu, Y., et al. (2022). AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale. PMLR 164.
14. Nair, S., Rajeswaran, A., Kumar, V., Finn, C., & Gupta, A. (2022). R3M: A Universal Visual Representation for Robot Manipulation. CoRL. arXiv:2203.12601.
15. Chi, C., et al. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Robotics: Science and Systems. arXiv:2303.04137.
16. Open X-Embodiment Collaboration. (2024). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. IEEE ICRA. arXiv:2310.08864.
17. Khazatsky, A., et al. (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. arXiv:2403.12945.
18. Chi, C., et al. (2024). Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. arXiv:2402.10329.
19. Octo Model Team, et al. (2024). Octo: An Open-Source Generalist Robot Policy. arXiv:2405.12213.
20. Black, K., et al. (2024). $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164.
21. Pertsch, K., et al. (2025). FAST: Efficient Action Tokenization for Vision-Language-Action Models. arXiv:2501.09747.
22. NVIDIA, et al. (2025). GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv:2503.14734.
23. Jang, E., et al. (2022). BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning. CoRL. arXiv:2102.02005.
24. Driess, D., et al. (2023). PaLM-E: An Embodied Multimodal Language Model. ICML. arXiv:2303.03378.
25. Lu, Y., et al. (2022). AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale. CoRL 2021 / PMLR 164. arXiv:2111.05424.
26. Shafiullah, N. M. M., et al. (2022). Behavior Transformers: Cloning k Modes with One Stone. NeurIPS. arXiv:2206.11251.
27. Xiao, T., et al. (2022). Masked Visual Pre-training for Motor Control. arXiv preprint. arXiv:2203.06173.
28. Chen, L., et al. (2026). ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control. arXiv primary preprint (2026-04-17). arXiv:2604.16677.
29. Huang, D., et al. (2026). VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Models. arXiv primary preprint (2026-05-28). arXiv:2605.29605.
30. Wang, Y., et al. (2026). Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization. arXiv primary preprint (2026-02-10). arXiv:2602.09722.
31. Ye, S., et al. (2026). World Action Models are Zero-shot Policies. arXiv primary preprint (2026-02-17). arXiv:2602.15922.
32. Kim, S., et al. (2025). FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation. arXiv primary preprint (2025-09-29). arXiv:2509.24241.