Chapter 1: Where Does Higher-Level Intelligence Connect? — The Task, Skill, and Control Contract Across Three Volumes

Overview

The vision-language-action model (VLA), world model, and agent in this volume are not a single brain that replaces every robot layer. They are upper-layer components that propose bounded tasks, skills, and actions from language and scenes. Below those proposals remain versioned skill application programming interfaces (skill APIs), feasibility projection, real-time controllers, independent evaluators, collision and force gates, watchdogs, protective stops, the emergency stop, and human approval authority.

S11 established the physical cell and first-motion contract. S12 established the execution spine from classical planning and learned policies to physical control. S13 does not teach those volumes again; it places broader data, language grounding, prediction, and long-horizon composition above the same spine. The deliverables of this chapter are not a model recommendation. They are a three-volume responsibility map, an S13 input-readiness checklist, and a proposal→projection→execution→evaluation receipt for tabletop assembly.

Learning goals and through-question

By the end of this chapter... you should be able to:

1. explain the input, output, cadence, clock, authority, and failure state of every layer from human intent to actuators;
2. distinguish what a VLA, world model, and agent may propose from the authority they do not own;
3. check the actual versions of inherited S11/S12 artifacts and close a gate instead of guessing missing values;
4. run one tabletop-assembly task with a classical baseline or upper-intelligence proposal while holding the lower contract fixed; and
5. separate model, planning, control, timing, safety, and operations failures and record an exact rollback unit.

Through-question: Given “pick up the blue part, insert it into the fixture in its new position, retreat once if it binds, then ask a human,” which component proposes what, who permits execution, and what evidence is required for promotion?

1.1 Put all three volumes on one responsibility map

1.1.1 Commands narrow downward; evidence and failure flow upward

Human intent / task specification
        ↓
VLA · world model · agent · memory/tool layer                 ← S13
        ↓ bounded task/skill proposal
Versioned skill API · finite-state machine · task planner     ← S12 contract
        ↓ bounded action/trajectory proposal
Inverse kinematics · motion planning · constraint projection  ← S11/S12
        ↓ feasible controller reference
Impedance · force · trajectory · vendor controller            ← S11/S12
        ↓
Robot · gripper/hand · sensors                                ← S11

Independent evaluator · watchdog · collision/force gates · protective stop ·
E-stop · responsible human cross every layer and are never inherited by a model.

Layered architectures and behavior trees expose the owner of each output and the boundary of each fallback, but their arbitration logic does not by itself establish the timing or physical safety of an assembly task. Brooks's physical mobile-robot layering, formal behavior-tree execution semantics, and CoSTAR's physical industrial-arm tasks support structure and fallback, not a universal manipulation-safety result (Brooks, 1986; Colledanchise & Ögren, 2018; Paxton et al., 2017). In particular, a mobile-robot suppression graph, tick semantics, or a human-authored task tree cannot establish the force limit or worst-case latency of a new contact task.

The arrows are more than message forwarding. “Insert the part” narrows to calls such as pick_part@2.1, move_to_precontact@1.4, and insert_bounded@3.0. A skill target is checked against the current robot model and scene geometry, then transformed into a trajectory or bounded contact reference before it reaches a controller. Joint state, contact events, evaluation results, rejection reasons, and stop state flow upward. Task and motion planning can couple symbolic choice to geometric feasibility, but it does not fill in unmodeled object state or contact (Kaelbling & Lozano-Pérez, 2011; Garrett et al., 2018).

1.1.2 Complete responsibility and authority table

One process may compute several rows; that does not merge their authorities. An agent may generate both a plan and a skill choice, while stop-reset authority remains with the responsible human and validated safety path. A shared platform and common logs can improve reproducibility, but hardware uniformity does not remove calibration, network, and operational drift (Paull et al., 2017).

1.2 What does upper intelligence output?

A VLA emits a task, skill, action token, action chunk, or trajectory proposal; that output does not inherit collision checking, force limiting, protective-stop, E-stop, or human authority. The direct basis is the model input/action interfaces and evaluation scopes reported for \pi_0 and OpenVLA, together with the separate authority boundary of runtime-assurance work (Black et al., 2024; Kim et al., 2024; Mehmood et al., 2022). This is an S13 responsibility synthesis across model evidence and simulated runtime assurance, not one paper's end-to-end tabletop-assembly system.

The three upper components do not occupy the same slot.

SayCan's grounding of language in skill affordances and Code as Policies' composition of robot functions are useful examples of connecting upper-layer semantics to real capabilities (Ahn et al., 2022; Liang et al., 2023). An affordance score is not a collision or contact-stability check, and generated code inherits the hidden side effects of every callable function. The first comparison is therefore not “which model is smartest?” It is “what candidate does this model emit through which interface, and which gate can reject it?”

1.3 The skill API turns a proposal into an execution contract

Generated plans and code require versioned skill preconditions, completion/failure status, timeout, and fallback. This complete field set is an S13 deployment synthesis. CoSTAR demonstrates behavior-tree execution and return status on real robots; Code as Policies exposes a prompted function/API namespace and documented code/API errors; AutoTAMP translates language into formal specifications and checks syntactic and semantic errors (Paxton et al., 2017; Liang et al., 2023; Chen et al., 2023). None reports the full version/timeout/fallback contract as one validated system. A curated API can still omit unsafe side effects or unobserved physical state, so the API list is not a safety guarantee.

At minimum, the insertion skill for the through-task carries this contract:

skill: insert_bounded
version: 3.0.2
inputs:
  part_id: scene object version
  fixture_id: scene object version
  approach_frame: calibrated frame id + calibration epoch
  max_depth: SI unit + validated bound reference
preconditions:
  - part_grasp_verified
  - scene_state_fresh
  - controller_mode == validated_contact_mode
  - collision_force_watchdog_ready
completion:
  - independent_evaluator == seated_within_tolerance
failures:
  - stale_state | infeasible | contact_limit | no_progress | evaluator_unknown
timeout: cell-validated duration reference
fallback:
  - retreat_bounded@1.6
  - request_human@1.1
authority:
  may_propose: [approach, contact_search, bounded_advance, retreat]
  may_not: [change_force_limit, clear_protective_stop, bypass_evaluator]

Neither the model nor this chapter invents numerical limits. The contract points to the actual S11 cell-risk and validation receipt. The executive records requested → preconditions_checked → projected → sent → accepted → executing → evaluated; it never compresses a missing middle state into success. Formal checking such as AutoTAMP's can inspect only encoded constraints. Fixture looseness, cable snag, and other out-of-model hazards remain separate tests.

1.4 Throughput is not a control deadline

Model throughput is not a controller deadline; observation age, queue state, tail latency, stale rejection, and failure action require end-to-end measurement. This is an S13 operational rule, not a result reported by Agia (2026). Agia’s primary evidence is limited to deployment-time runtime monitoring of action-chunk temporal consistency and task progress (including a VLM-monitor latency measurement); it does not specify the complete clock, queue, and stale-command contract or a portable deadline for this cell (Agia, 2026). Set actual limits on the current hardware under the current load.

Every action proposal needs at least these timestamps:

• t_event: when the observation physically occurred;
• t_arrival: when it reached the model-input process;
• t_infer_start, t_infer_end: the inference interval;
• t_queue_in, t_queue_out: residence in the execution queue;
• t_sent, t_accepted, t_executed: downstream transmission, acceptance, and execution;
• valid_until: the last instant at which the proposal may be used; and
• clock_domain, sync_epoch, uncertainty: identity and uncertainty of clock conversion.

A low mean inference time is insufficient if rare tail events exceed the action horizon, or if a fast model repeatedly consumes stale observations. For action chunks, define whether a replacement flushes the remainder, how overlap is blended, and which validated hold or retreat state is entered on disconnect. A stale proposal is not executed because the next one looks promising; it receives a distinct rejection code while its raw output is retained.

1.5 Through-example: trace one tabletop-assembly trial

The default cell is a 6/7-DoF arm with a parallel gripper and wrist/scene cameras. The fixed task recognizes part and fixture, grasps, transports collision-free, stops before contact, places or inserts under bounded contact, detects failure, and retreats, retries, asks a human, or aborts. S13 adds paraphrased instructions, new objects and layouts, occlusion, sequential composition, and failure recovery.

Step 1 — Convert the instruction into a task specification

The agent structures “insert the blue part into the fixture on the right” as a target object, fixture, goal state, forbidden region, and permitted recovery count. If “right” lacks a reference frame or two blue parts are visible, the result is ambiguous; the system rejects execution and asks a human. Chapter 5 will cover open-vocabulary and spatial grounding. Here, the deliverable is the contract at its output boundary.

Step 2 — Propose skills

A VLA or agent proposes locate → grasp → transport → stop_precontact → insert_bounded → verify. A world model may rank two approach candidates, but an attractive predicted video does not confer execution authority. The task executive checks that every skill ID and version is on the allowlist and that preconditions, termination, and timeout are complete.

Step 3 — Project through the S12 execution spine

The grasp and approach target pass IK and collision-aware planning against the current scene generation. A planning failure is not recorded as a model failure. The system preserves a classical trajectory baseline through the pre-contact stop, then switches to a validated contact-control mode for insertion. A candidate generator such as information-theoretic model-predictive control still depends on model, cost, sampling, and compute-time assumptions; it is not automatically a high-rate servo guarantee (Williams et al., 2017).

Step 4 — Separate proposal, projection, transmission, and acceptance/execution

Store the raw VLA proposal, constraint-filtered projected, controller-bound sent, controller-confirmed accepted, and state-confirmed executed records separately. If two differ, preserve the reason and transformation loss. Without this split, “the model failed” cannot be distinguished from “a good proposal became stale and was correctly rejected.”

Step 5 — Close the loop with independent evaluation and recovery

Insertion completion is judged by an independent evaluator reading pose, depth, contact, and visual state—not by the model's textual answer. On contact_limit, perform a validated retreat; on no_progress, retry only within the approved count. If occlusion makes the evaluator unknown, do not count success; request a human. AutoRT demonstrates upper-model orchestration across a large robot task collection, but its robots, tasks, supervision, and evidence do not substitute for this cell's safety receipt (Ahn et al., 2024).

Step 6 — Receipt what changed and what stayed fixed

Begin comparisons by changing one field, proposal_source. When comparing a classical skill sequence, an S12 learned policy, and a VLA skill proposal, hold robot, calibration, scene, skill API, projector, controller, evaluator, safety configuration, and initial distribution fixed where possible. If multiple factors changed, mark the trial as confounded rather than attributing a gain to the model.

1.6 Independent safety remains outside the model

Simplex architectures, control barrier functions, and predictive safety filters apply only under explicit plant, state, constraint, timing, and recoverability assumptions (Mehmood et al., 2022; Ames et al., 2017; Wabersich & Zeilinger, 2021). Their theorems and simulated examples support filtering a candidate or switching to a safety controller; they are not a complete tabletop-cell risk assessment or certification.

“There is a safety module” is not evidence of independence. Exercise it:

1. If the model process stops or the network disconnects, the watchdog enters a validated hold, retreat, or stop.
2. NaN, out-of-range, wrong-frame, wrong-version, and stale proposals are rejected before the controller.
3. Collision, force, workspace, and protective-stop events are logged on a path separate from model telemetry.
4. Transfer to the safety controller completes before the recoverable set is lost, including worst-case sensing, compute, and switching delay.
5. An authorized human resets a protective stop or E-stop; no model may bypass or approve the reset.
6. The independent evaluator does not rely solely on the model output or on one representation with the same failure cause.

SOTER separates a high-performance controller from a verified-safe controller through runtime assurance, but its guarantee depends on the monitor model and recoverable region, and switching can reduce task performance (Desai et al., 2019). If a safety gate triggers often, do not remove it as a performance nuisance. Diagnose proposal distribution, projection, state estimation, and gate configuration separately.

1.7 S13 input-readiness checklist and promotion gates

A missing or incompatible frame, calibration, controller, evaluator, or rollback artifact closes the promotion gate rather than inviting the model to guess (Desai et al., 2019; Mehmood et al., 2022). This closed-gate rule is an S13 deployment synthesis of two runtime-assurance sources, not a tabletop-assembly procedure reported verbatim by either paper.

Before using verified, assign one of these states to each item:

• missing: the required artifact, device, or owner does not exist;
• not measured: it exists, but the value has not been measured in this configuration;
• not exercised: it is configured, but its fault/recovery path has not been tested;
• incompatible: version, unit, frame, mode, or clock does not match; or
• verified: it has identifiable evidence and an owner.

Promotion proceeds through static schema check → recorded replay → offline evaluation → simulation → shadow execution → human-reviewed bounded trial. Passing one stage is not evidence for the next. Even a physical-hardware result is not “deployment ready” without human exposure, force events, aborts, and recovery denominators.

1.8 Common failure-diagnosis table

Minimum metrics go beyond success rate. Record trials and successes by task and initial distribution, proposal rejection, feasibility rejection, human intervention, force/collision/stop events, median and upper-tail observation age and end-to-end latency, queue discard, recovery attempts/successes/aborts, evaluator-unknown rate, per-trial cost, and exact rollback time. Split generality into five axes: new instruction on the same robot, new scene/object, new task, new embodiment, and longer composition.

Every trial receipt includes:

• the task card and reset/exclusion rules;
• a version tuple for robot, sensors, calibration, controller, and safety configuration;
• raw observations and proposal → projected → sent → accepted/executed logs;
• skill transitions, evaluation, failure, intervention, and recovery events;
• hashes for model, data, code, skill API, and inference environment;
• evidence stage, generalization axis, human exposure, and cost; and
• approver, deployment time, prior approved tuple, and restore-test result.

1.9 A bounded Codex implementation prompt

The prompt below builds an offline contract validator inside a repository. It confers no authority to move a robot or connect to a device or service.

Goal: implement a read-only CLI that validates the S13 proposal receipt and
readiness report for tabletop assembly.

Allowed scope:
- Read and edit only repository schemas/, fixtures/, tests/, and new validator code.
- Do not connect to a robot, ROS graph, vendor controller, camera, or network service.
- Do not invent force, velocity, workspace, or timing limits. Check only validated
  references already present in an input receipt.

Required inputs:
- task_card, system_tuple, skill_manifest, proposal_receipt, evaluator_receipt,
  safety_receipt, rollback_tuple

Required validation:
1. Check schemas and every version/hash/frame/unit/clock_domain field.
2. Require proposal, projected, sent, accepted, and executed as distinct events.
3. If any status is missing/not_measured/not_exercised/incompatible, close the
   gate with exit code 2 and the exact gate ID.
4. Reject stale, NaN, out-of-range, unknown skill/version, and wrong-frame inputs.
5. Reject any input that assigns safety authority or E-stop reset to a model/agent.
6. Preserve raw log values and attach validator version to every derived judgment.

Tests:
- Create one valid fixture and one fixture for each failure above.
- Run deterministic unit tests only. Never label hardware evidence as passed.

Output:
- readiness_report.json: status, evidence path, owner, checked_at for each gate
- validation_summary.md: changed files, tests run, pass/fail, unresolved items
- On failure, stop without automatic relaxation or bypass.

Codex itself belongs on the authority map. It may propose and test validator code; it may not infer site limits, acquire safety authority, or open a failed gate. If generated code can call skills, apply the same schema checks, allowlist, sandbox, deterministic replay, and human review.

1.10 Limits of the comparison and bridge to Chapter 2

This map is a design discipline, not a performance report for one completed system. Layering exposes ownership but can make a large behavior tree difficult to version and audit. Planning does not see hazards outside its modeled geometry and skills. Safety filters lose their guarantee when assumptions about state estimation, plant, solver timing, or recoverability fail. Recent VLA and deployment-reliability preprints still need independent replication and long-term operations evidence. Above all, no source in this chapter validates the complete S13 tabletop-assembly stack or transfers final command and safety authority to a model.

The book therefore does not begin with model size or a demonstration video. Chapter 2 examines the data and action representations that enter this map: what robot episodes, video, language, and synthetic data can share; how degrees of freedom, frames, and control modes survive cross-embodiment mixing; and where continuous actions, chunks, tokens, latent actions, and skill calls connect. If the Chapter 1 readiness gate is closed, Chapter 2's data mixture does not begin.

Further reading

• For research context on coding agents and manipulation interfaces, see Terry's Coding Agents for Manipulation. This is a reader crosslink, not claim evidence.
• For contact-rich global-planning context, see Global Planning for Contact-Rich Manipulation. This is also a reader link, not evidence for this chapter's claims.

Supplementary evidence map

The following verified primary-source groups come from the independent S13 ledger. They document search coverage without changing existing claims or denominators.

• architecture and action interfaces: See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models, World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy, Large Video Planner Enables Generalizable Robot Control, RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation — source-specific tasks, data, and evidence stages remain separate.
• state, reward, and failure tracking: Reward Machine Inference for Robotic Manipulation, Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphs, A Survey on Vision-Language-Action Models for Embodied AI — source-specific tasks, data, and evidence stages remain separate.
• generative and recovery lineages: Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation, GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation, Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts — source-specific tasks, data, and evidence stages remain separate.

References

1. Brooks, R. A. (1986). A Robust Layered Control System for a Mobile Robot. IEEE Journal on Robotics and Automation.
2. Colledanchise, M., & Ögren, P. (2018). Behavior Trees in Robotics and AI: An Introduction. CRC Press.
3. Paxton, C., Hundt, A., Jonathan, F., Guerin, K., & Hager, G. D. (2017). CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision. IEEE ICRA.
4. Kaelbling, L. P., & Lozano-Pérez, T. (2011). Hierarchical Task and Motion Planning in the Now. IEEE ICRA.
5. Garrett, C. R., Lozano-Pérez, T., & Kaelbling, L. P. (2018). FFRob: Leveraging Symbolic Planning for Efficient Task and Motion Planning. The International Journal of Robotics Research.
6. Desai, A., Ghosh, S., Seshia, S. A., Shankar, N., & Tiwari, A. (2019). SOTER: A Runtime Assurance Framework for Programming Safe Robotics Systems. IEEE/IFIP DSN. DOI: 10.1109/DSN.2019.00027.
7. Mehmood, U., Sheikhi, S., Bak, S., Smolka, S. A., & Stoller, S. D. (2022). The Black-Box Simplex Architecture for Runtime Assurance of Autonomous CPS. NASA Formal Methods. DOI: 10.1007/978-3-031-06773-0_12.
8. Ames, A. D., Xu, X., Grizzle, J. W., & Tabuada, P. (2017). Control Barrier Function Based Quadratic Programs for Safety Critical Systems. IEEE Transactions on Automatic Control.
9. Wabersich, K. P., & Zeilinger, M. N. (2021). A Predictive Safety Filter for Learning-Based Control of Constrained Nonlinear Dynamical Systems. Automatica.
10. Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023). Code as Policies: Language Model Programs for Embodied Control. IEEE ICRA.
11. Chen, Y., Ding, Y., Branicky, M. S., & Zhang, Z. (2023). AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers. IEEE ICRA.
12. Black, K., Brown, N., Driess, D., et al. (2024). $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164.
13. Kim, M. J., Pertsch, K., Karamcheti, S., et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model. arXiv:2406.09246.
14. Ahn, M., Dwibedi, D., Finn, C., et al. (2024). AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents. arXiv:2401.12963.
15. Agia, C. (2026). Deployment-Time Reliability of Learned Robot Policies. arXiv:2603.11400v1.
16. Paull, L., Tani, J., Ahn, H., et al. (2017). Duckietown: An Open, Inexpensive and Flexible Platform for Autonomy Education and Research. IEEE ICRA.
17. Williams, G., Aldrich, A., & Theodorou, E. (2017). Information-Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving. IEEE Transactions on Robotics.
18. Ahn, M., Brohan, A., Brown, N., et al. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. CoRL.
19. Lee, B., et al. (2026). See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models. arXiv primary preprint (2026-07-13). arXiv:2607.11498.
20. Liu, X., et al. (2026). World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy. arXiv primary preprint (2026-02-06). arXiv:2602.06508.
21. Chen, B., et al. (2025). Large Video Planner Enables Generalizable Robot Control. arXiv primary preprint (2025-12-17). arXiv:2512.15840.
22. Liu, S., et al. (2024). RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation. arXiv primary preprint (2024-10-10). arXiv:2410.07864.
23. Baert, M., et al. (2024). Reward Machine Inference for Robotic Manipulation. arXiv primary preprint (2024-12-13). arXiv:2412.10096.
24. Behrens, T., et al. (2024). Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphs. arXiv primary preprint (2024-11-28). arXiv:2411.19162.
25. Ma, Y., et al. (2024). A Survey on Vision-Language-Action Models for Embodied AI. IEEE Transactions on Neural Networks and Learning Systems (Early Access), 2026. arXiv:2405.14093.
26. Bu, Q., et al. (2024). Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation. arXiv primary preprint (2024-10-10). arXiv:2410.08001.
27. Cheang, C., et al. (2024). GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation. arXiv primary preprint (2024-10-08). arXiv:2410.06158.
28. Chen, H., et al. (2024). Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts. arXiv primary preprint (2024-09-06). arXiv:2409.03966.