Chapter 8: How Can We Trust Generality? — Evaluation, Contamination, Failure, Safety, and Operations

Overview

The phrase “general-purpose robot” erases cells from an evaluation table. Moving a part within the familiar assembly cell, following a novel instruction about a novel object, transferring to another arm and gripper, and composing many skills over a long horizon support different claims. Pooling them into one success rate hides what transferred, what was exposed during training, what the feasibility layer rejected, and where a person rescued the run. In this chapter, generality is an evaluation contract: it states what remained fixed, what changed, the exposure history, denominators, failures, and evidence stage.

S11 supplies the physical cell, sensor and clock identities, frames and calibration generation, controllers, collision and force bounds, watchdog, protective stop, E-stop, and accountable human. S12 supplies versioned task and skill contracts, proposal → projected → sent → accepted/executed lineage, feasibility projection, classical and learned baselines, an independent evaluator, and the complete rollback tuple. Chapter 7 adds distinct authority and receipts for planning, memory, tools, skill calls, and recovery. This chapter does not replace any of them. It builds the independent evidence ledger that determines the bounded scope of a claim.

Three questions organize the chapter. First, how should evaluation separate same-robot/task variation, novel scenes, objects, and instructions, cross-embodiment transfer, and long-horizon composition? Second, how should it report intervention, unsafe proposals, feasibility rejection, latency, recovery, calibration drift, distribution shift, and cost alongside success? Third, how can evaluator and safety-supervisor failures remain independent of the model and of each other?

After reading this chapter, you will be able to... - design separate regimes for the same robot and task, new scenes, objects and instructions, cross-embodiment transfer, and long horizons; - record intervention, rejection, tail latency, recovery, calibration drift, shift, and cost on the same lineage as success; - distinguish training/evaluation overlap, contamination, near duplicates, selective reporting, simulation-only evidence, and bounded hardware evidence; - use calibration, out-of-distribution detection (OOD), selective prediction, and conformal prediction without granting them motion authority; and - apply a foundation-policy evaluation card, failure taxonomy, and operational stop/rollback contract to tabletop assembly.

The experiment question is: with task success, initial-state distribution, observation/action meaning, skills, controllers, safety bounds, and evaluator fixed, does the foundation policy or agent beat the classical and S12 learned baselines in each declared regime? Does it retain that advantage after unsafe proposals, intervention, rejection, latency, failed recovery, and operating cost are restored to the record?

1. Mental model: an evidence lattice, not one average

A diagram of model → success rate drops task exposure, embodiment, time, execution gates, and service state. The useful mental model is a lattice with four dimensions: regime × evidence stage × execution lineage × failure layer. Evidence in one cell does not automatically fill another.

versioned model · data · task · robot · evaluator · safety contract
                              ↓ exposure, duplicate, split audit
 same robot/task | new scene | new object | new instruction | new embodiment | long horizon
                              ↓ check fixed and changed axes per regime
       replay → offline → simulation → shadow → bounded hardware
                              ↓ preserve lineage in every trial
 proposal → projected/rejected → sent → accepted → executed → evaluated
                              ↓ record failure, time, and cost with success
 intervention | unsafe proposal | latency | recovery | drift | stop | handoff

Independent transverse rails: sensor-based evaluator · collision/force supervisor
                              watchdog · protective/E-stop · human authority

The horizontal axis states what was novel. The vertical axis states how directly the evidence touches the current cell. Execution lineage reveals candidates rejected or modified before success. Failure layers prevent an instruction-grounding error and a force event from collapsing into the same failure label. An evaluation card must join all four axes for each run.

The evaluator judges the task goal from sensor evidence. The safety supervisor enforces collision, force, speed, workspace, and deadline constraints during motion. Combining them makes an error in goal recognition propagate into a safety verdict. They also share a common cause if both consume the policy’s scene representation or the same remote vision service. The evaluator must not use the policy’s narrative as ground truth. The supervisor must not use model uncertainty as permission to clear a stop.

2. Split generality into six regimes

In-distribution, cross-scene, cross-object, cross-instruction, and cross-embodiment results support different claims ^{[21] [25] [1]}. Positive transfer across multiple robots and open-world examples matters, but its scope remains bounded by each source’s tasks, data, adaptation, denominators, and possible overlap. Novelty or exposure left unstated for one regime cannot be inferred from another.

The running assembly task needs a sixth regime: long-horizon composition. Executing one skill on a novel object differs from completing identify → grasp → transport → pre-contact stop → insert → verify → recover. A small error changes the next skill’s initial state, while memory, tools, remote services, and replanning introduce additional dependencies.

2.1 “Novel object” has several meanings

An object absent from robot training video may have appeared in Internet pretraining. It may be a recolored instance of the same CAD asset, or the same episode may have crossed the split under a new filename. Static vision studies already showed that models can learn dataset-specific cues and lose performance across datasets ^[1]. Sequential robot logs add possible cues such as camera pose, fixture scratches, an operator’s hand, reset order, or controller sound. The static-image result motivates the audit but does not quantify those robot-specific effects.

The unit of exposure is therefore an event lineage, not a file. Trace the original episode, clipped segment, augmentation, transcoded video, language paraphrase, synthetic rendering, retrieval document, and memory summary. If exposure cannot be established, label it unknown_exposure, not clean. A new-object claim should separately mark category, physical instance, geometry lineage, visual asset, pretraining, fine-tuning, retrieval, and memory exposure.

2.2 Cross embodiment starts with compatibility, not success

Open X-Embodiment provides an important common format for multi-institution, multi-robot data and cross-embodiment research ^[21]. Dataset-level normalization does not necessarily preserve controller mode, compliance, force and tactile meaning, gripper semantics, timing, or safety limits. The first output on a new robot should be a compatibility audit with compatible, adapted, incompatible, or not_exercised, not a pooled success percentage.

Moving from the reference arm and parallel gripper to another arm requires new camera extrinsics, workspace, joint/end-effector mapping, action rate, gripper convention, speed/force limits, and contact-controller identification. A converter may accept the same high-level skill while producing different trajectories and contact. Record adaptation data and physical exposure as costs. “Zero-shot” must not erase hidden calibration, manual selection, prompt adjustment, or failed trials.

3. Matched axes and the foundation-policy evaluation card

Match the evaluation unit before making a leaderboard. A policy tested on 20 new objects, another on 50 familiar objects, and a third on 1,000 simulated scenes do not supply comparable success rates. If a required field in the following table is absent, mark the comparison not comparable.

The foundation-policy evaluation card is an index into per-run receipts, not a model-card summary:

evaluation_card: s13.foundation-policy-eval/v1
subject: {model, checkpoint, adapter, prompt, decoder, policy_head}
contracts: {task, robot, observation_action, skill_api, controller, evaluator, safety}
regime: {same_robot_task, new_scene, new_object, new_instruction, cross_embodiment, long_horizon}
exposure: {train, adaptation, retrieval, memory, near_duplicate, internet_pretraining, verdict}
trial_denominator: {planned, initialized, proposed, projected, sent, accepted, executed, evaluable}
evidence_stage: replay|offline|simulation|shadow|bounded_hardware
outcomes: {success, partial, fail, unknown, intervention, human_assisted}
proposal_safety: {unsafe, infeasible_rejected, stale_rejected, schema_rejected}
timing: {observation_age, inference_p50_p95_p99, queue, projection, deadline_miss}
recovery: {detected, stopped, retreated, retried, replanned, handed_off, recovered}
shift: {detector, calibration_version, predeclared_axes, drift_events}
operations: {remote_outage, local_fallback, fleet_scope, privacy, copyright, cost}
rollback: {model, data, prompt, skill, controller, evaluator, calibration, service_versions}

The denominator chain matters. Using only executed removes unsafe or infeasible proposals rejected before execution. Counting every rejection as task failure can punish a correctly functioning gate. Report proposal quality and executed outcome separately and connect both with proposal_id. Also retain the distance between proposal and projected action; a policy that “succeeds” only because a lower layer frequently rewrites its output is not equivalent to one that proposes feasible actions.

4. Promote evidence stages without substituting them

Replay recomputes proposals and rejections from logged events without exposing future outcomes. Offline evaluation measures components such as instruction grounding, object recognition, action prediction, or uncertainty. Simulation permits controlled scene and physics variation and repeatable fault injection. Shadow execution uses live sensors, remote services, and queues but sends no robot command. Only a bounded hardware trial includes the current cell’s friction, cables, calibration, latency, and contact.

SIMPLER studies simulation as a proxy for evaluating real-world manipulation policies, including relationships under distribution shifts, but it does not replace bounded hardware evidence ^[22]. The relationship is conditional on the tasks, assets, policies, and simulator. It cannot be extended to untested deformables, contact, sensors, controllers, or operational faults. Keep simulated and physical results under different evidence_stage values on the same card.

Domain randomization provides physical examples of transfer from randomized visual or dynamics conditions, while leaving contact and dynamics outside the randomized range unresolved ^{[11] [15]}. Adapting randomization with real experience still depends on hardware exposure and selected trajectory statistics ^[16]. A simulation pass therefore means eligible_for_shadow, not safe_for_hardware.

Each stage preserves counterexamples found earlier. Excluding a simulation-only failure as “unrealistic” requires a predeclared exclusion rule and evidence. A shadow output that misses its deadline is an operational failure even though no command was sent. If a person repositions a part during a hardware trial, the result is human_assisted, not autonomous success.

4.1 Read long horizons as conditional stages

VLABench exposes difficulties in language-conditioned long-horizon manipulation and unseen-object/instruction settings, but its result is benchmark-specific simulation evidence rather than a hardware release receipt ^[24]. It remains bounded by the MuJoCo/dm_control benchmark's workflows, objects, and evaluation definitions and cannot establish contact, timing, stop, or recovery behavior in the running tabletop cell.

A long episode needs both a final outcome and a stage vector. Record entry, completion, failure, and dwell time for recognize, grasp, transport, precontact, insert, verify, and recover, then connect the first failure to subsequent actions. A failure observed at insertion may originate in grasp pose or stale scene state. Multiplying stage success rates is also unsafe because failures are dependent and recovery changes the state distribution.

Do not split one long run into several short successes. Preserve episode_id and attempt_family_id. A run completed by a person after the retry budget is separate from autonomous completion. Budget a fault family so renaming a repeated error cannot create unlimited retries.

5. Audit contamination, duplicates, and selective reporting first

Contamination is broader than an identical evaluation filename in a training manifest. It includes alternate encodings of the same video, adjacent frames, clipped subtrajectories, augmented images, instruction paraphrases, synthetic renderings of the same asset, retrieval entries, and memory summaries. If the full Internet pretraining corpus is unknown, do not assert clean separation.

Use four layers. First, detect exact duplicates by dataset, episode, asset, source-video hash, DOI, or canonical identifier. Second, find near duplicates using perceptual signatures and temporal alignment. Third, make grouped splits by object, background, camera, operator, fixture, and instruction meaning. Fourth, record when evaluation content enters retrieval, memory, or human demonstration during the run. Automatically flagged pairs need sampled human review, with threshold, false-positive, and false-negative limits retained.

Selective reporting creates another leakage path. Choosing seeds, checkpoints, prompts, cameras, objects, resets, retry budgets, or exclusions after seeing the outcome can improve the visible average. Evidence from deep-RL evaluation shows that few-run point estimates can yield unstable comparisons; small, non-independent hardware denominators demand at least as much caution ^{[13] [18]}. Pre-register the matrix, preserve every attempt, and report intervals, performance distributions, and regressions.

Every report needs four counts together: planned, initialized, excluded, and evaluable. Exclusion codes should distinguish predeclared hardware faults, operator abort for safety, corrupt logging, and forbidden post-hoc exclusion. A missing failure video or raw log is incomplete evidence, not a zero event count.

5.1 Keep comparisons blind where feasible

The operator who knows which model is active can unconsciously choose easier placements, intervene earlier, or interpret partial seating more favorably. Randomize model order and initial states through a recorded scheduler, blind outcome annotation where possible, and let a second reviewer inspect disagreements. Blinding does not remove hardware drift, so interleave baselines and candidates rather than running all baseline trials on a fresh morning and candidates after calibration and fixture wear have changed.

Report every mid-evaluation change. Recalibration, camera cleaning, firmware update, prompt repair, service migration, or fixture replacement starts a new block. Do not pool blocks unless the analysis model and strata were declared. Operational improvement is valuable, but it must not masquerade as model generality.

6. Separate calibration and distribution shift from safety authority

An uncertainty score can trigger a question; it is not a safety certificate. Confidence, ensemble disagreement, selective abstention, and conformal sets optimize different targets under different assumptions. Calibration of static class probability does not imply calibration of a structured action, contact hazard, or long-horizon failure probability.

Temperature scaling, deep ensembles, selective prediction, and conformal prediction address calibration, risk-coverage, or marginal coverage, but they cannot authorize robot motion and may degrade under distribution shift ^{[7] [9] [17] [19]}. Marginal coverage can conceal dangerous subgroups or states. Exchangeability, calibration data, output structure, and the chosen risk must be checked for sequential manipulation rather than assumed.

Maximum softmax probability is a useful OOD baseline whose detection behavior varies across dataset pairs; high scores can remain far from training data ^[8]. Selective classification can lower error by reducing coverage, but it does not define the robot fallback after abstention ^[6]. In assembly, abstention must route to a physically valid stop_before_contact, retreat_to_precontact, or request_human state—not an unspecified “hold last command.”

Calibrate separately by regime. Do not carry a threshold fitted on the familiar cell unchanged into a new camera, object, instruction, or robot. Examine reliability diagrams, risk-coverage curves, negative log likelihood or Brier score, outcomes after rejection, subgroup failures, and drift over time, not only expected calibration error. Version and roll back the policy, encoder, calibrator, and threshold together.

6.1 Diagnose even when confidence stays high

Use a fixed order:

1. Sensors and time: frame loss, observation age, clock regression, camera and force-sensor health.
2. Calibration and geometry: frames, extrinsics, fixture/gripper references, residual trends.
3. Exposure and shift: new asset, background, instruction, operator, and service version.
4. Policy output: uncertainty, ensemble disagreement, action discontinuity, precondition violation.
5. Execution gates: projection rejection, collision/force limit, stale command, queue, controller acceptance.
6. Evaluation and safety: independent sensor evidence, unknown verdict, watchdog and stop events.

Failure can arrive before a model score falls; sensors, representations, and evaluator may share a common cause. Stop on predeclared force, rejection, tail-latency, recovery, or calibration-residual thresholds even when the shift detector is quiet. Conversely, an alert should move the cell to a safe state rather than induce a risky diagnostic motion.

7. Red-team the system and classify the failure

Red teaming is not only pixel perturbation. Include ambiguous or prohibited instructions, forged tool results, poisoned memory, stale observations, duplicate calls, remote timeouts, evaluator spoofing, frame/unit confusion, frozen sensors, and safety-service loss. Attack the proposal–check–execute–evaluate chain, not merely the policy.

One adversarial VLA study reports up to a 100% task-success reduction within its simulated suite under its tested attacks and models ^[23]. The number is bounded by the attack, model, action discretization, tasks, and simulation denominator; it is not a universal physical-robot failure rate. The result motivates a distinct adversarial regime and a test of whether lower gates reject unsafe proposals—not the claim that every VLA fails at the same rate.

A red-team exercise need not create a dangerous physical motion to succeed. Replay and shadow mode can produce unsafe proposals and verify projection and supervisor rejection. Necessary hardware tests stay within approved low-energy bounds, physical barriers, explicit stop criteria, and direct human supervision. Do not repeatedly expose hardware merely to prove the E-stop works.

The counterexample bundle persists. Replay it whenever model, prompt, data, tool, calibration, evaluator, or supervisor changes. An attack still counts if the policy resists it but the queue or recovery manager fails. Report where the candidate was stopped, rejection latency, and false rejection—not detection rate alone.

8. Privacy, copyright, and governance are operating metrics

Tabletop-cell video can capture faces, voices, hands, badges, screens, and surrounding documents. Privacy begins with purpose, minimization, consent and withdrawal, access, retention, location, and deletion evidence—not just post-hoc blurring. Sending frames or instructions to a remote service adds transmission scope, retention or training terms, subcontractors, region, encryption, and outage behavior to the evaluation card.

Copyright and data governance must distinguish access to a paper from permission to reuse its figure, code, weights, dataset, or operator video. Combining public datasets does not create an umbrella license. Review license, consent, provenance, and deletion duties for each constituent artifact. An asset with uncertain rights is unknown_license and remains blocked from external transfer, new training, and publication.

Governance can block release without changing task success. If a deletion request cannot be traced through derived clips, embeddings, caches, memory, and checkpoints, rollback is incomplete. Where audit retention conflicts with minimization, restrict raw-media access and separate event/hash records from content. This engineering pattern is not legal advice; accountable security, privacy, and legal owners must interpret jurisdiction and contracts.

9. Test remote inference and fleet failure

Remote inference provides compute while adding network transit, service queues, retry behavior, unannounced model changes, and authentication failure to the control path. Fleet operation lets one checkpoint reach many cells and turns a bad cache or shared observation fault into a common-cause failure. Mean latency and mean success hide these tails.

Link observation event time, service arrival, inference start/end, policy use, projection, send, and acceptance using one trace ID. Report p50, p95, and p99, timeouts, cancellation, late-response disposal, reconnect queue semantics, and duplicate calls. A useful action that arrives after its deadline is a failure and must not be sent. Outage fallback depends on physical state: controlled free-space stopping, pre-contact stopping, or verified force/impedance hold or retreat may differ. One global fallback cannot cover them all.

Release evidence should record unsafe proposals, feasibility-projection rejection, intervention, force events, tail latency, stale commands, outage, recovery, cost, and rollback ^{[26] [20]}. Runtime monitoring and safe-controller switching support keeping detection and fallback outside the learned policy, but their evidence is conditional on the stated failure categories and bounded dynamics/timing. An unreported operational field is not reported, not zero.

Fleet promotion starts with a small canary cohort. Stratify robot, camera, calibration, controller, and network conditions. Do not change the shared model and local assets simultaneously. On anomaly, block new requests, bring current contact to a safe state, and restore the exact prior tuple. Rolling back only the server while retaining a new local adapter or calibrator creates a new configuration, not the previous system.

9.1 Count cost without turning it into a single price

Cost includes operator labeling and supervision, reset time, damaged parts and fixtures, robot wear, accelerator time, network egress, remote-service calls, storage, privacy review, and opportunity cost of cell downtime. Keep financial cost, human time, energy, and hardware exposure as separate columns before any weighted summary. A cheaper method that transfers hidden work to operators is not cheaper under the same boundary.

Associate cost with outcomes and regimes. New-embodiment adaptation may improve success while requiring extra calibration and hardware hours; long-horizon autonomy may reduce direct teleoperation but increase intervention diagnosis. Report both. An evaluation budget is also a safety exposure budget: stop when repeated failures add no diagnostic information.

10. Engineer failure independence

Separate processes are not automatically independent. A policy and evaluator sharing the same visual encoder, training split, scene graph, or remote service can misidentify the same object. A safety supervisor sharing the GPU, operating system, clock, network, or power supply can become unavailable when the policy fails.

Disagreement is information. The evaluator may report successful seating while the safety supervisor reports near-limit force. Preserve both and stop promotion. Neither verdict overwrites the other. One component may summarize another’s status but cannot write its verdict or clear its stop.

Black-Box Simplex supplies an architecture in which advanced and baseline controllers can remain black boxes while runtime reachability determines switching ^[20]. Its guarantees still require bounded dynamics and timing; tabletop contact needs separate evidence for those assumptions. The transferable lesson is architectural: learned proposals remain upstream of an independent switch and fallback.

11. Tabletop evaluation workflow and gates

Evaluation begins by freezing denominators and stop conditions, not by collecting a success reel.

1. Freeze contracts: version task, initial distribution, success predicate, prohibitions, robot, calibration, skills, controller, evaluator, safety, and human owner.
2. Audit exposure: compare training, adaptation, retrieval, and memory against exact, near, and grouped evaluation overlap; emit clean, known_overlap, or unknown.
3. Register matrix: predeclare six regimes, changed axes, denominators, seeds/checkpoints/prompts, exclusions, statistical plan, and cost limits.
4. Replay and offline: prevent future leakage; replay instruction, proposal, rejection, uncertainty, and evaluator outputs; build the counterexample bundle.
5. Simulation and adversarial: inject shift and unsafe proposals; record rejection, false rejection, and latency by gate.
6. Shadow: use live sensors, remote services, and queues while asserting sent_count=0.
7. Bounded hardware: open one regime and one change at a time under low-energy limits, direct approval, and immediate stop criteria.
8. Independent review: evaluation, safety, and operations owners—not the policy owner—inspect raw logs, exclusions, cost, counterexamples, and rollback.
9. Promote or narrow: expand one boundary only after passing; otherwise restore the exact tuple and add the failure to replay.

A gate may return missing, not_measured, not_exercised, or incompatible in addition to pass/fail. All four close physical promotion. A new robot without force sensing has not_measured force events, not zero. An untested outage has not_exercised reliability, not a pass.

11.1 Trace one assembly trial end to end

The instruction is: “Insert the new gray bracket into the right fixture; if it does not fit, retreat once and ask me.” Start with same robot/task, then make separate cards for new object only, paraphrased instruction only, and their combined shift. Do not pool the combined condition into either single-axis regime.

The policy proposes grasp → transport → precontact → insert. If projection rejects the first insertion pose, record proposed=1, projected_rejected=1, and sent=0. A second candidate after re-observation stays in the same trial lineage. If insertion completes after a person moves an occluder, report human_assisted=1 separately from autonomous success.

If the evaluator sees successful seating while the force supervisor reports a near-limit event, preserve task and safety outcomes and halt promotion. Diagnose part tolerance, calibration drift, contact control, proposed pose, and evaluator threshold separately. A success video does not erase the force event.

12. Operational stop and rollback contract

Stopping is not merely killing the model process. It must block new proposals, invalidate queued commands, transition current physical contact safely, hand authority to a person, seal evidence, and restore a complete prior tuple. Protective and emergency stops belong to devices and safety layers; neither model nor remote operator clears them.

12.1 Immediate stop triggers

• collision, force, speed, or workspace limit exceedance; watchdog or protective-stop event;
• repeated evaluator–supervisor disagreement, frozen sensor, or suspected common cause;
• stale command sent, execution after deadline, undefined queue/reconnect semantics;
• unauthorized skill, tool, code, or data access; privacy or rights-boundary violation;
• unsafe proposal, rejection, intervention, failed recovery, or tail latency crossing a preregistered threshold;
• calibration, frame, robot, controller, evaluator, or supervisor version mismatch; or
• missing raw logs, denominators, rollback tuple, or discovered evaluation contamination.

12.2 Stop sequence

1. Refuse new proposals and remote jobs.
2. Invalidate all pre-send queue entries and late responses.
3. Let the independent safety path classify free space, pre-contact, contact, or unknown.
4. Use an approved local controller to decelerate, hold safely, retreat, or hand off according to that state.
5. If a protective or E-stop fired, keep it latched until an authorized person inspects the cause.
6. Seal model, data, service, skill, controller, evaluator, calibration, approval, and raw-log identities.
7. Restore the last known-good tuple and block hardware until replay and shadow validation pass.

The rollback tuple includes at least model/checkpoint, adapter, system prompt/policy, decoder, dataset manifest, retrieval index, memory snapshot, skill registry, projection layer, controller, evaluator, safety supervisor, calibration, robot firmware, remote service, feature flags, and approval ID. Restoring only some members creates a new system requiring evaluation.

12.3 Resume conditions

Classify the cause, reproduce the counterexample, replay the repaired version against the fixed baseline and complete failure bundle, and obtain independent evaluation and safety approval. Service recovery or restored confidence alone is insufficient. A contact or protective-stop event also requires physical inspection and calibration checks. Resume in shadow and a small canary cell, never fleet-wide.

13. Counterevidence and limits

This procedure is not a complete safety proof. Calibration and shift results developed on classification do not directly cover sequential contact and operational latency ^{[7] [17]}. Conformal marginal coverage may be inadequate under adaptive, non-exchangeable operation or in hazardous subgroups ^[19]. Ensemble members can share data and architecture and fail together ^[9].

Simulation proxies bring control and repeatability, but their correlation can change with assets, physics, and sensors ^[22]. The large adversarial degradation is bounded to a particular simulated setup ^[23]. A long-horizon benchmark reveals composition and language difficulty without supplying contact or operations receipts ^[24]. Runtime assurance depends on correct bounds and completing its check before the deadline ^[20].

An independent evaluator remains vulnerable to sensor faults, wrong success predicates, and calibration error. A safety supervisor enforces defined bounds but cannot discover every task hazard. Privacy, copyright, and governance obligations vary by jurisdiction and contract. Most importantly, no source in the packet validates the complete S13 tabletop stack. Combining component papers does not create a system-level guarantee.

The evaluation card therefore preserves uncertainty rather than erasing it. Do not estimate missing quantitative fields. Distinguish not reported from zero, unknown from failure, and not exercised from pass. This bounded language is more useful than an unqualified generality claim.

14. Metrics and artifacts to retain

• Generality: planned/executed/evaluable denominators by regime; complete, partial, fail, unknown; baseline difference and intervals; combined-shift conditions.
• Exposure: exact, near, and grouped duplicates; training/adaptation/retrieval/memory exposure; unknown Internet pretraining; exclusion ledger.
• Proposal/execution: unsafe proposals, schema/stale/feasibility rejection, projection delta, send/accept/execute lineage.
• Time: observation age; inference, queue, projection, and end-to-end p50/p95/p99; deadline miss, late disposal, throughput.
• Recovery: detection, stop, retreat, re-observation, retry, replan, handoff, recovery success, harmful recovery, and time.
• Physical/safety: collision, force, speed, watchdog, protective/E-stop; evaluator–supervisor disagreement; sensor and calibration drift.
• Operations/cost: remote outage, local fallback, fleet scope, human time, reset, compute, network, hardware exposure and damage.
• Governance: consent, access, retention, deletion, personal-data transfer, license, provenance, redistribution, and change approval.

Required artifacts are evaluation_matrix, exposure_manifest, duplicate_audit, trial_registry, execution_lineage, evaluator_receipt, safety_receipt, calibration_report, shift_report, red_team_bundle, failure_taxonomy, operations_log, cost_ledger, governance_card, stop_receipt, rollback_tuple, and resume_approval. Each carries version, digest, timestamp, owner, and source-evidence locator. A success video and model explanation are supporting material, never substitutes.

15. Bounded Codex implementation prompt

Goal
- Build independent evaluation cards for six generalization regimes from recorded tabletop episodes.
- Compare policy/agent candidates with classical and S12 baselines under matched contracts and denominators.
- Produce a failure taxonomy, adversarial bundle, and stop/full-rollback receipt.

Context
- Read S11 robot/sensor/clock/frame/calibration/controller/safety identities.
- Preserve S12 task/skill API, proposal→projected→sent→accepted/executed,
  independent evaluator, evidence stages, and rollback tuple.
- Treat Chapter 7 planning/memory/retrieval/tool/recovery receipts as read-only inputs.

Constraints
- Send no physical command; assert shadow_hardware_command_count=0.
- Never pool the six regimes, evidence stages, or assisted and autonomous outcomes.
- Expose no future event during replay.
- Do not mark unclear overlap or exposure as clean.
- Never let model score, uncertainty, or self-evaluation bypass collision/force,
  watchdog, protective/E-stop, or human authority.
- Never turn missing/not measured/not exercised/incompatible into zero or pass.
- Do not transmit or train on assets with unclear privacy or rights.

Work
1. Validate task, robot, data, evaluator, safety, and rollback contracts; stop if incomplete.
2. Build exact/near/group duplicate and training/adaptation/retrieval/memory exposure ledgers.
3. Register fixed/changed axes, denominators, exclusions, and faults for all six regimes.
4. Replay baseline and candidate with identical timestamped input and evaluator.
5. Count proposal/projected/rejected/sent/accepted/executed/evaluated separately.
6. Inject ambiguous instructions, new assets, stale observations, tool/remote outage,
   duplicate calls, calibration drift, evaluator spoofing, and force events.
7. Report success, intervention, unsafe proposals, rejection, p50/p95/p99,
   recovery, cost, and governance.
8. Apply preregistered stop thresholds and replay restoration of the full rollback tuple.

Completion
- Every result maps to regime, exposure, denominator, evidence stage, lineage, and failures.
- Evaluator and safety supervisor have separate versions, sensors, authority, and fault tests.
- Improved and degraded axes, counterexamples, exclusions, unmeasured fields, and cost are reported.
- Simulation is not represented as physical release evidence.
- Stop with a request for a separate human approval card if hardware evidence is required.

Safety
- Outputs are evaluation, rejection, and recovery candidates plus receipts, never motion authority.
- Protective/E-stop clearing and final resume remain with named people and devices.
- During service failure, only verified local fallback appropriate to contact state is allowed.

Manufacturing Cell Checkpoint

Task/data: Are the success predicate, initial states, six regimes, training/adaptation/retrieval/memory exposure, and duplicate audit versioned? Execution/logging: Does the ledger retain proposal → projected/rejected → sent → accepted/executed → evaluated denominators and event/arrival/use times? Metrics: Does it report intervention, unsafe proposals, rejection, tail latency, recovery, calibration drift, shift, and human/compute/hardware cost alongside success? Safety: Are evaluator and supervisor failure-independent of the policy, with no model score able to clear stops or authorize motion? Operations: Are privacy, rights, remote outage, fleet canary, full rollback, and named resume owners resolved?

Any missing, not measured, not exercised, or incompatible answer closes hardware promotion. Same-robot success, simulation correlation, low uncertainty, a clean-sounding split name, a success video, or a healthy remote service cannot fill the gap.

Relation to Prior Surveys

This chapter does not reteach the S11 safe cell or S12 hybrid classical/learned execution spine. It fixes them as the lower boundary of evaluation. Further reader context is available in ForceVLA, π0, DexForce, Octo, Coding Agents for Manipulation, and RoboClaw. These are reader crosslinks, not claim evidence.

16. Hand off release evidence, not a demonstration, to Chapter 9

Chapter 9 completes the three volumes in one tabletop cell. This chapter hands over evaluation_card, the fixed/changed axes for six regimes, exposure and duplicate audits, baseline/candidate denominators, execution lineage, evaluation and safety receipts, failure and adversarial bundles, latency/recovery/cost/governance ledgers, and the exact stop, rollback, and resume contract—not a best success rate.

Chapter 9 must retain these axes while comparing the classical skill baseline, S12 policy, VLA skill proposal, optional world model, and agent composition. When adding one component, keep task, control, evaluation, and safety contracts fixed. Do not promote complexity if it fails to beat the baseline, worsens unsafe proposals, latency, intervention, or cost, or cannot be independently replayed.

What to Learn Next

Chapter 9 places specified task → classical skill baseline → S12 learned policy → VLA skill proposal → optional world model → agent skill composition on the same cell and evaluation card. This chapter’s stop and rollback contract is the entrance to every promotion stage, not an appendix. Completion in the final chapter means connecting the three-volume version lineage, failure evidence, bounded physical trial, human approval, and maintenance receipt—not producing a success montage.

Supplementary evidence map

The following verified primary-source groups come from the independent S13 ledger. They document search coverage without changing existing claims or denominators.

• physical reasoning and touch: LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines, Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation, InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning, The Art of Imitation: Learning Long-Horizon Manipulation Tasks from Few Demonstrations — source-specific tasks, data, and evidence stages remain separate.
• long-horizon planning: CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models, Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D, Subgoal Diffuser: Coarse-to-fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulation — source-specific tasks, data, and evidence stages remain separate.
• transfer and generated constraints: HACMan++: Spatially-Grounded Motion Primitives for Manipulation, RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation, Open-World Task and Motion Planning via Vision-Language Model Generated Constraints — source-specific tasks, data, and evidence stages remain separate.

References

1. Torralba, A., & Efros, A. A. (2011). Unbiased Look at Dataset Bias. CVPR. DOI:10.1109/CVPR.2011.5995347.
2. Deisenroth, M. P., & Rasmussen, C. E. (2011). PILCO: A Model-Based and Data-Efficient Approach to Policy Search. Artificial Intelligence. DOI:10.1016/j.artint.2011.02.004.
3. Finn, C., Goodfellow, I., & Levine, S. (2016). Unsupervised Learning for Physical Interaction through Video Prediction. NeurIPS. arXiv:1605.07157.
4. Pinto, L., & Gupta, A. (2016). Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours. IEEE ICRA. arXiv:1509.06825.
5. Finn, C., & Levine, S. (2017). Deep Visual Foresight for Planning Robot Motion. IEEE ICRA. arXiv:1610.00527.
6. Geifman, Y., & El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. NeurIPS. arXiv:1705.08500.
7. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML. arXiv:1706.04599.
8. Hendrycks, D., & Gimpel, K. (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. ICLR. arXiv:1610.02136.
9. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS. arXiv:1612.01474.
10. Paull, L., et al. (2017). Duckietown: An Open, Inexpensive and Flexible Platform for Autonomy Education and Research. IEEE ICRA. arXiv:1711.05438.
11. Tobin, J., et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IEEE/RSJ IROS. arXiv:1703.06907.
12. Williams, G., Aldrich, A., & Theodorou, E. (2017). Information-Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving. IEEE Transactions on Robotics. DOI:10.1109/TRO.2017.2769099.
13. Henderson, P., et al. (2018). Deep Reinforcement Learning that Matters. AAAI. arXiv:1709.06560.
14. Nagabandi, A., Kahn, G., Fearing, R., & Levine, S. (2018). Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning. IEEE ICRA. arXiv:1708.02596.
15. Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. IEEE ICRA. arXiv:1710.06537.
16. Chebotar, Y., et al. (2019). Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience. IEEE ICRA. arXiv:1811.11758.
17. Ovadia, Y., et al. (2019). Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. NeurIPS. arXiv:1906.02530.
18. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., & Bellemare, M. (2021). Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS. arXiv:2108.13264.
19. Angelopoulos, A. N., & Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511.
20. Mehmood, U., Sheikhi, S., Bak, S., Smolka, S. A., & Stoller, S. D. (2022). The Black-Box Simplex Architecture for Runtime Assurance of Autonomous CPS. NASA Formal Methods. DOI: 10.1007/978-3-031-06773-0_12. arXiv:2102.12981.
21. Open X-Embodiment Collaboration (2024). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. IEEE ICRA. arXiv:2310.08864v9.
22. Li, X., et al. (2024). Evaluating Real-World Robot Manipulation Policies in Simulation. arXiv:2405.05941v1.
23. Wang, T., et al. (2024). Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics. arXiv:2411.13587v4.
24. Zhang, S., et al. (2024). VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks. arXiv:2412.18194v1.
25. Physical Intelligence, Black, K., et al. (2025). $π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054v1.
26. Agia, C. (2026). Deployment-Time Reliability of Learned Robot Policies. arXiv:2603.11400v1.
27. Cherian, A., et al. (2024). LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines. arXiv primary preprint (2024-11-12). arXiv:2411.08027.
28. Giudici, G., et al. (2024). Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation. arXiv primary preprint (2024-12-03). arXiv:2412.02644.
29. Han, M., et al. (2024). InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning. arXiv primary preprint (2024-05-30). arXiv:2405.19758.
30. Hartz, J. O. v., et al. (2024). The Art of Imitation: Learning Long-Horizon Manipulation Tasks from Few Demonstrations. arXiv primary preprint (2024-07-18). arXiv:2407.13432.
31. Huang, H., et al. (2024). CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models. arXiv primary preprint (2024-03-13). arXiv:2403.08248.
32. Huang, H., et al. (2024). Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D. arXiv primary preprint (2024-01-22). arXiv:2401.12046.
33. Huang, Z., et al. (2024). Subgoal Diffuser: Coarse-to-fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulation. arXiv primary preprint (2024-03-19). arXiv:2403.13085.
34. Jiang, B., et al. (2024). HACMan++: Spatially-Grounded Motion Primitives for Manipulation. arXiv primary preprint (2024-07-11). arXiv:2407.08585.
35. Kuang, Y., et al. (2024). RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation. arXiv primary preprint (2024-07-05). arXiv:2407.04689.
36. Kumar, N., et al. (2024). Open-World Task and Motion Planning via Vision-Language Model Generated Constraints. arXiv primary preprint (2024-11-13). arXiv:2411.08253.