Chapter 7: What Should a Robot Agent Own? — Planning, Memory, Tools, and Skill APIs

Overview

The value of a robot agent is not that it writes an impressive plan in natural language. Its value is that it decomposes a long task into short, verifiable steps, calls only versioned skill APIs, observes what execution actually produced, and stops, re-observes, or replans when evidence disagrees with expectation. The agent is neither a torque controller nor a collision judge. Its output is an execution candidate that must pass the inherited execution contract before it becomes a command.

S11 contributes robot, gripper, and sensor identities; clocks, frames, and calibration generations; controllers; collision and force limits; the watchdog, protective stop, E-stop, and accountable human. S12 contributes versioned task and skill contracts, feasibility projection, IK, motion planning and contact control, proposal → projected → sent → accepted/executed lineage, an independent evaluator, evidence stages, and the complete rollback tuple. This chapter adds a distinct identity, authority, and record for planning, memory, retrieval, tools, code generation, recovery, and human approval.

Chapter 6 handed forward predicted outcomes, uncertainties, residuals, and rejection reasons for candidates. The new question is: who may compose those candidates into multiple skills, which gates turn a generated plan into reproducible skill calls, and what may recover automatically before a human must take over? The answer is mostly about boundaries. Memory cannot approve a plan. Retrieved text cannot grant tool authority. Generated code cannot manufacture a new robot command.

After reading this chapter, you will be able to... - decompose a long tabletop-assembly instruction into a skill graph with preconditions, completion conditions, and failure states; - choose among behavior trees (BTs), finite-state machines (FSMs), task and motion planning (TAMP), and LLM/VLM planners on matched axes; - assign separate authority to planning, memory, retrieval, tools, code, skill execution, recovery, and human approval; - validate generated plans, text, and code through schemas, allowlists, sandboxing, deterministic replay, and lower safety gates; and - build a failure/recovery state machine that distinguishes bounded retry, retreat, human handoff, and stop.

The experiment question is: with the assembly task, initial-state distribution, skill versions, controller, evaluator, safety limits, and denominators fixed, does the agent reduce incomplete skills and human intervention relative to a fixed baseline plan? Does it do so without increasing unauthorized calls, stale plans, repeated failures, force events, or irreproducible recovery?

1. Mental model: a layer of proposals and receipts, not a layer of thought

Drawing the agent as one box conflates the planner, memory, retriever, code runner, and safety supervisor. Instead, separate every transformation by input, output, authority, and expiry. A natural-language instruction becomes a versioned task specification. That specification becomes a graph made only of allowlisted skills. A proposed call passes precondition and freshness checks before reaching lower planning and projection. Execution returns an event that either supports or falsifies the plan.

human instruction + task contract + fresh scene receipt
                          ↓ semantic, prohibition, completion checks
             plan candidate + evidence + assumptions + expiry
                          ↓ schema, skill allowlist, version checks
                  ordered/graph skill-call candidates
                          ↓ precondition, state, idempotency checks
          TAMP/IK/motion planning/projection/contact control
                          ↓ collision, force, deadline, human approval
                    sent → accepted/executed
                          ↓ independent evaluation and failure events
             continue | re-observe | retreat | alternate | human | stop

Always independent: watchdog · protective stop · E-stop · final human authority

An LLM or VLM may structure the goal, propose a skill sequence, or summarize a result. It may not alter a skill implementation, lower a safety limit, or clear a protective stop. A world model may supply expected candidate outcomes, but it is not an execution approver. Every agent plan needs a plan_id, model, prompt, tool and memory versions, input event time, expiry, and explicit assumptions. A missing field is missing, not success, and closes the hardware gate.

The useful mental model is not “think, then act.” It is propose → check → project → execute → independently judge → recover. Eloquence of the reasoning trace matters less than the ability to replay the same call graph from the same inputs and versions. When a narrative disagrees with physical evidence, independent sensor events and evaluator output win.

2. Choosing a planner on matched axes

BTs, FSMs, classical task planning, and model-based planners need not be mutually exclusive. TAMP can find a feasible manipulation sequence, a BT can supervise execution and fallback, and a model can translate an ambiguous instruction into a candidate specification. Compare them by state representation, feasibility handling, exposed failure, reproducibility, change cost, and authority—not by impressions such as “flexibility” or “reasoning.”

An FSM makes event-triggered transitions explicit, but object and error combinations can multiply states. A BT composes fallback paths through the success, failure, and running contract, but the same tree can behave differently if tick rate and internal node state are not recorded ^[4]. CoSTAR provides a physical collaborative-robot example combining vision and BT-based instruction, while retaining human-authored task structure and platform-specific perception and manipulation failures ^[3].

Classical planning prioritizes consistency of stated predicates and effects over linguistic fluency. TAMP additionally asks whether symbolic pick(part) has a grasp and collision-free path. Hierarchical planning “in the now” can avoid instantiating every future continuous state, but it cannot detect contact omitted by the model or geometry ^[2]. The integrated TAMP review is a useful taxonomy; as a synthesis, it is not itself quantitative evidence for this assembly cell ^[6].

TAMP and behavior trees expose preconditions, feasibility, execution status, and fallback, making them auditable baselines for model-based agents ^{[2] [6] [4]}. Their symbols, geometry, and trees can nevertheless remain incomplete, so using these structures does not guarantee physical safety or task completeness. Their simulation, real-system, and synthesis evidence stages must also remain distinct.

An LLM/VLM planner can map a language goal to candidate sequences inside a fixed skill vocabulary. SayCan provides a physical-robot example that combines language likelihood with learned skill value to select skills ^[8]. Its result is conditional on its robot, skill library, instruction distribution, and denominator. The multiplicative score is not a calibrated safety probability and does not transfer to this assembly cell. A model planner should supply candidates between predefined skills and checkers, not remove either one.

3. Agent authority table: separate read, propose, execute, and approve

The riskiest design gives one service identity permission to write memory, retrieve documents, run code, call robot skills, and judge its own success. Neither root cause nor privilege escalation is then isolatable. The following table fixes what each component owns and what it must never own.

Planning, memory, tools, code, skills, recovery, and approval require separate identities, logs, allowlists, replay procedures, and lower safety gates ^{[18] [20] [7]}. AutoRT combines higher-level models and human supervision in large-scale real-robot orchestration while reporting that hallucinated objects, generalization failure, and motion blur can propagate failures. Black-Box Simplex supports separating an unverified advanced proposal from runtime-assured control, but its guarantees depend on bounded dynamics and timing and indirect non-contact cases. Complete assurance against poisoned memory and irreversible tool effects remains unproven.

4. The skill-call schema: a firewall between language and robot commands

A skill call is not merely a function name plus numbers. It binds task, scene, calibration, skill and controller versions; units and frames; preconditions; completion and failure conditions; timeout; duplicate-call semantics; fallback; and approval class. The following is a schema illustration, not executable code.

skill_call_schema: s13.skill-call/v1
call_id: call-0187
plan_id: plan-0042
task_contract: tabletop-assembly@7.1
skill:
  name: insert_part
  version: 2.3.1
  implementation_digest: sha256:...
scene_receipt:
  observation_id: obs-991
  event_time: "2026-07-16T10:22:31.420+09:00"
  max_age_ms: 120
  frame_tree: cell-tf@11
  calibration: wrist-cam@8
arguments:
  part_id: bracket-A
  fixture_id: jig-2
  target_pose: {frame: fixture, unit: m-rad, value: [...]}
  contact_limits: {force_N: 12, torque_Nm: 1.2}
preconditions:
  - part_grasp_verified
  - fixture_clear
  - precontact_pose_reached
completion:
  evaluator: insertion-eval@4
  predicate: seated_and_force_released
failure_policy:
  retry_budget: 1
  reversible_until: contact_depth_2mm
  on_failure: retreat_to_precontact
authorization:
  risk_class: contact_bounded
  human_approval_id: approval-73
deadline: "2026-07-16T10:22:32.000+09:00"
idempotency_key: task7-step5-attempt1

Validation goes beyond field presence. It checks that insert_part@2.3.1 is currently registered, arguments satisfy declared ranges and units, observation age is valid, frame and calibration generations match, and human approval covers the risk class and time interval. A numeric force_N above the cell limit is still rejected. Repeating an idempotency key must return the previous result rather than physically execute again.

The planner may propose the schema but cannot invent an implementation_digest or approval ID; registry and approval services supply them. Unknown fields are rejected rather than silently accepted. An explicit converter and regression replay are required before a call from an older schema can reach a newer skill.

4.1 Code generation is a narrower tool

Code as Policies shows that a language model can generate hierarchical programs over a human-curated function namespace and compose physical robot tasks ^[12]. The intermediate program can be inspected and functions reused. It also inherits every hidden assumption, unit, side effect, and error-handling behavior of those functions.

Code as Policies generates programs over curated functions and therefore does not justify arbitrary code execution in a robot cell ^{[12] [3]}. Its physical evidence is conditional on human-designed functions and interfaces, and it does not generally test sandbox escape or irreversible side effects. Generated code should therefore begin as a pure planning transform in isolation, with no network, shell, device, secrets, or write authority.

A suitable first target is a pure scene_receipt → typed_skill_calls transform. Its output passes schema validation, static analysis, property tests, fixed-corpus replay, and resource limits. Generated code never invokes the physical skill directly; the skill broker validates it again. A failure is not patched live and resent to hardware. A new version must replay the same held-out corpus and counterexample bundle.

5. Separate translation, formal checking, and physical feasibility

Translating an instruction into a planning language, checking logical consistency, and checking whether the robot can execute are different problems. If “insert the red bracket without touching the blue cable” loses avoid(cable-blue) during translation, a logical solver may faithfully satisfy the wrong specification. A correct logical plan may still lack a grasp or collision-free path. A valid path may still violate force, speed, or timing limits.

AutoTAMP separates LLM translation, formally checked task planning, and motion feasibility, while its formal checker covers only encoded constraints ^{[13] [2]}. Its main evidence is simulation and assumes correct scene graphs, operators, and constraints; passing formal checks is not physical collision or force validation. Translation omission, logical inconsistency, geometric infeasibility, and control-limit violation need distinct rejection codes.

The minimum workflow has six gates.

1. Intent gate: check target, goal state, prohibitions, completion, ambiguity, and need for human clarification.
2. Plan gate: use allowlisted skills and verify that every precondition/effect chain is consistent.
3. State gate: check observation age, object identity, frames, calibration generation, and hidden state.
4. Feasibility gate: check grasp, IK, collision-free path, projection, and pre-contact stop.
5. Operations gate: check inference latency, deadline, queue, stale rejection, idempotency, and service fallback.
6. Safety/approval gate: check collision, force and speed bounds, watchdog, protective stop, human approval, and rollback target.

Each gate returns more than pass: it can return reject, unknown, stale, incompatible, or not_measured. Unknown is never treated as true, timeout is never success, and a planner explanation never overrides evaluator failure. proposal_id, projection_id, command_id, and execution_id connect the gates so the faulty transformation remains identifiable.

6. Tabletop walkthrough: compose a long instruction from skills

The instruction is: “Find the red bracket on the occluded shelf, place it into the left slot of the jig, and if it does not seat, realign only once and then ask me.” The fixed task remains part and fixture recognition, grasp, collision-free transport, pre-contact stop, bounded insertion, failure detection, retreat, retry, human request, or stop. The agent adds only the sequence and conditional branches.

6.1 Produce the plan candidate

The goal becomes seated(bracket-red, jig-left), the prohibition becomes no_contact(cable-blue), the retry budget is one, and terminal failure becomes request_human. If the frame for “left” is absent, the agent asks rather than plans. If occlusion leaves object identity uncertain, it proposes reobserve(viewpoint-2) as the first skill. A retrieved successful plan is evidence from the past, not a current scene fact.

The candidate graph is reobserve → localize → grasp → transport → precontact → insert → evaluate. If evaluate=success, it completes. A recoverable_alignment_failure permits exactly one retreat → reobserve → realign → insert branch. Force exceedance, identity conflict, calibration incompatibility, or protective stop does not permit automatic retry; the system holds safe and hands off. A budget applies to a failure family so renaming the same fault cannot create infinite attempts.

6.2 Read a typed result and replan

A skill returns structured evidence rather than one Boolean.

skill_result_schema: s13.skill-result/v1
call_id: call-0187
execution_id: exec-551
status: failed
failure_class: recoverable_alignment
started_at: "..."
ended_at: "..."
observation_ids: [obs-991, obs-1004]
proposal: {pose: [...], force_N: 10}
projected: {pose: [...], force_N: 10}
sent: {trajectory_id: traj-77}
accepted_executed: {accepted: true, executed_prefix_ms: 340}
evaluator:
  id: insertion-eval@4
  outcome: not_seated
safety_events: []
residuals: {lateral_mm: 1.8, peak_force_N: 9.6}
recovery_allowed: [retreat_to_precontact, reobserve]

The agent reads the failure class and allowed recovery set, then proposes the next call. peak_force_N below its limit is one condition for considering retry, not evidence of success. If the evaluator returns unknown, the agent cannot overwrite it with a video description. Retreat is itself a versioned skill and passes collision, force, and deadline checks again.

6.3 Falsify in shadow mode first

Replay held-out episodes event by event. The planner sees only information available at each timestamp, never future outcomes. Compare a baseline FSM/BT against the agent on skill choice, missing preconditions, rejection, recovery choice, and latency. Then use simulation and shadow mode to record projection and branches without sending a command. shadow_hardware_command_count must be zero.

A bounded physical trial opens only under a separate human approval card. Begin with low speed and force, one object, one fixture, and one failure branch. Do not change controller, force limits, evaluator, or initial distribution at the same time as the planner. Preserve failure, rejection, intervention, and the exact rollback tuple before selecting a success video.

7. Memory, retrieval, tools, and code are separate capabilities

7.1 Memory is a lineage-bearing event store

Separate three layers. Episodic memory contains immutable observations, calls, outcomes, and safety events. Semantic memory contains approved skill descriptions, task rules, and scene concepts. Procedural memory contains versioned skills and verified plan templates. A model-written failure summary is derived data, never a replacement for the event. It retains source events, generating model, timestamp, trust scope, and expiry.

REFLECT summarizes robot experiences to explain failures and modify subsequent attempts ^[15]. In AI2THOR simulation, it reports about 80% correction-planning success for both execution and planning failures. Its separate real-world UR5e set contains 30 human-teleoperated failure scenarios and evaluates explanation and localization, not physical retry completion.

A successful REFLECT retry does not prove that its failure explanation was causally correct ^[15]. A summary may omit critical state or hallucinate a cause. Preserve the explanation beside independent sensor events, execution lineage, and evaluator verdict, and measure explanation accuracy separately from recovery success.

Memory cannot grow without policy. BUMBLE uses prior failures for long-horizon mobile manipulation but identifies computationally intractable growth as a limitation ^[19]. Compression lowers storage and retrieval cost but may erase a rare, safety-critical failure. Define retention, compression, deletion approval, and reproducible index versions.

BUMBLE reports explicit success/failure categories from 70 trials, but its building-wide mobile platform and growing memory are an advanced branch, not the default single-arm tabletop cell ^[19]. Its denominators and interfaces are platform-specific, so its outcome cannot be compared directly with assembly success here. It motivates failure categorization and memory operations, not a performance guarantee for this task.

7.2 Retrieval supplies material, not authority

The retriever returns approved skill documentation, task contracts, and prior receipts with provenance. Every result needs document ID, version, author, valid cell, creation time, and trust class. Retrieving a “20 N insertion” note from another robot does not alter the current cell limit. External web text and user notes are untrusted material, not commands.

Poisoning tests include a document with embedded malicious instructions, an obsolete skill, another frame convention, retired calibration, and a memory containing only successes. The gate closes if the agent prioritizes document prose over the active contract. Evaluate not only relevant-document recall but the fraction of wrong retrievals that influence a proposed call.

7.3 Tools carry side-effect classes

A read-only scene query and a robot move are not one category of “tool.” Classify tools as pure_read, bounded_write, physical_reversible, physical_irreversible, or safety_critical, then set per-actor allowlists and approval requirements. Toolformer shows self-supervised API selection for text tools, but likelihood filtering does not guarantee tool correctness or authorization ^[11]. ReAct interleaves reasoning, action, and observation to update a plan, but its text environments omit physical timing, contact, and irreversible actions ^[17].

The broker normalizes arguments and checks timeout, retry semantics, side effects, idempotency, and result schema. A network failure becomes tool_unavailable, not an empty result. Write and physical tools are not retried automatically. Even read results carry creation time and maximum age so stale data cannot masquerade as a current observation.

7.4 Generated code does not enter the skill library directly

New code is written only to a proposal store. It passes static analysis, dependency locks, sandbox tests, resource limits, fixed-input replay, property tests, and human review before becoming a promotion candidate. A promoted skill is fixed as skill_name@version + implementation_digest + schema_version. A live patch is a new version and changes approval and rollback targets. Generated code receives no arbitrary shell, dynamic package installation, external network, driver device, or safety I/O.

8. Failure and recovery state machine

Recovery is not “try again.” It is a state transition conditioned on reversibility, whether more observation can disambiguate the fault, whether contact changed the part, remaining exposure budget, and human approval scope.

RECEIVED
  ├─ missing/ambiguous/prohibited → NEEDS_CLARIFICATION → HUMAN_REVIEW
  └─ valid → PLANNED
PLANNED
  ├─ schema/version/allowlist failure → REJECTED
  ├─ stale observation → REOBSERVE → PLANNED
  └─ valid → PROJECTED
PROJECTED
  ├─ IK/collision/force/deadline failure → INFEASIBLE → REPLAN | HUMAN_REVIEW
  ├─ approval required/missing → AWAITING_APPROVAL
  └─ pass → EXECUTING
EXECUTING
  ├─ protective stop/E-stop/limit → SAFE_STOP → HUMAN_REVIEW
  ├─ recoverable failure → RETREATING → REOBSERVE → REPLAN
  ├─ unknown/irreversible failure → HOLD_SAFE → HUMAN_REVIEW
  └─ skill complete → EVALUATING
EVALUATING
  ├─ success → NEXT_SKILL | COMPLETE
  ├─ failure + budget remains → REPLAN
  ├─ unknown → REOBSERVE | HUMAN_REVIEW
  └─ budget exhausted → HOLD_SAFE → HUMAN_REVIEW

The planner cannot clear SAFE_STOP. Resuming after an E-stop follows the device procedure and requires the named human’s on-site check. RETREATING is not assumed universally safe: with contact, cables, and surrounding parts, retreat may worsen the fault. It is a versioned skill with its own feasibility check.

Retry budget counts more than calls. It includes physical exposure, cumulative force and time, part-state change, and human intervention. A failure_family_id prevents relabeling the same fault to evade budget. If the same state produces the same plan again, detect the deterministic loop and hand off. A recovered task retains its original failure and intervention cost in the denominator.

9. From symptom to cause: diagnostic order

Diagnosis starts with raw observations and event times, call/projection/send/accept/execution lineage, controller and evaluator results, and safety events—not the planner’s narrative. Next inspect planning inputs, tool results, and memory summaries. Use the model explanation last as a hypothesis. This ordering can reveal that “the model confused the object” was actually calibration-generation mismatch or a stale queue.

Deterministic replay requires more than freezing model output. Bind system prompt, model/tokenizer, sampling, tool responses, retrieval index, memory snapshot, schema and skill registries, time, random state, and planning budget. If a remote model cannot reproduce its response, preserve the original output and separate decision replay from model recall testing.

10. Evidence, counterevidence, and open gaps

Evidence stages differ. TAMP and BT sources support explicit state, precondition, status, and fallback structure. SayCan, Code as Policies, Inner Monologue, CoSTAR, and SayPlan provide physical or physical-inclusive examples of language-conditioned skill selection, code composition, feedback, and scene graphs ^{[8] [12] [9] [3] [16]}. REFLECT executes correction plans in AI2THOR simulation; its physical UR5e set evaluates explanation and failure localization only ^[15]. Their robots, skill catalogs, scenes, human engineering, and denominators differ; their success numbers do not belong in one leaderboard.

AutoRT reports long-running physical orchestration across multiple buildings and many robots ^[18]. That is scale evidence, not independent safety or recovery evidence for default tabletop assembly. BUMBLE’s categories are likewise platform-specific mobile-manipulation evidence ^[19]. A 2026 embodied-agent study reports overconfidence, task refusal, inefficient recovery from failed tools, and ambiguous-instruction errors, but its evidence is simulation and requires independent replication ^[20].

Counterexamples are design inputs. Symbolic models omit friction, occlusion, and deformation. A large BT can lose version and hidden-state auditability. Language planners invent objects or skills and erase negative constraints. A scene graph can propagate a structured perception error. Memory can reinforce a wrong success verdict; retrieval can supply stale or poisoned documents; generated code can compose hidden API side effects. Human approval can become a blanket waiver if plan changes and expiry are not checked.

No single primary source validates this chapter’s complete tabletop stack or transfers robot-command and safety authority to a model. Observation age, tail latency, queue/stale semantics, intervention, force events, recovery denominator, and complete rollback are often not reported. We record them as not reported, never infer them. Recent preprints are design candidates; promotion needs independent replication and local operational receipts.

11. Metrics and artifacts to retain

Success rate alone cannot show whether an agent improved. Retain at least:

• Planning: goal/prohibition preservation, feasible graph rate, unregistered-skill proposal rate, missing preconditions, plan depth, and replans.
• Time: observation age; median and p95/p99 planning, retrieval, tool, and projection latency; deadline misses and stale-output rejection.
• Tools/code: call correctness, schema rejection, calls by side-effect class, sandbox failures, replay agreement, promotion and rollback counts.
• Memory: source-event linkage, retrieval recall and contamination, stale-memory use, summary-sensor contradiction, and failure retention after compression.
• Execution: counts at proposal/projected/sent/accepted/executed, projection delta, skill complete/fail/unknown denominators, and controller rejection.
• Recovery: type-specific detection, retreat/re-observation/replan outcome, repeated faults, time to handoff, correct and harmful recovery.
• Safety/operations: unsafe proposals, collision/force events, protective and E-stops, human intervention, remote outage, hardware exposure, and cost.

Required artifacts are task_contract, agent_authority_table, skill_registry, skill_call_schema, plan_receipt, tool_receipt, memory_snapshot, projection_receipt, execution_receipt, evaluator_receipt, recovery_trace, approval_card, and rollback_tuple. Each has version, hash, timestamp, and actor. Video is supporting evidence, not a substitute for those identities.

Promotion proceeds through replay → offline counterexamples → simulation → shadow mode → human-reviewed bounded trial. Preserve separate denominator and failures at every stage. Do not promote the complex agent if it does not beat the baseline FSM/BT or classical planner, worsens latency/rejection/intervention, or cannot be replayed.

12. Bounded Codex implementation prompt

Goal
- Convert long tabletop instructions in recorded episodes into versioned skill-call graphs.
- Produce separate receipts for planning, memory, retrieval, tools, code, recovery, and approval.
- Compare baseline FSM/BT and agent candidates in replay and shadow mode.

Context
- Read S11 robot/sensor/clock/frame/calibration/controller/safety identities.
- Preserve S12 task/skill API, projection, evaluator, evidence-stage, and rollback contracts.
- Inputs are held-out episodes, read-only scene receipts, and an approved skill registry.
- Outputs are plan/skill-call/recovery candidates, never physical robot commands.

Constraints
- Call no skill, tool, file, network, shell, or device outside its allowlist.
- Reject unknown fields, unit/frame/version mismatches, and stale observations.
- Test generated code only as a pure transform in a no-network, no-write, no-device sandbox.
- Freeze system prompt, model, sampling, tool responses, index, and memory snapshot.
- Never bypass evaluator/collision/force/watchdog/protective-stop/E-stop/human authority.
- Never fill missing/not measured/not exercised/incompatible as success.

Work
1. Validate task_contract and skill registry; stop if incomplete.
2. Extract goal, prohibitions, completion, ambiguity, and retry budget from each instruction.
3. Generate baseline FSM/BT and agent plans from the same timestamped information.
4. Record plan_id/call_id/version/deadline/idempotency/preconditions for every call.
5. Feed historical events one at a time and record continue/reobserve/retreat/replan/human/stop.
6. Test future leakage, poisoned retrieval, stale observation, tool outage, and repeated-failure cases.
7. Compare schema rejection, feasibility rejection, latency, recovery, and intervention on fixed denominators.
8. Send no command and assert shadow_hardware_command_count=0.

Completion
- Agent authority table, call/result schemas, and failure/recovery state machine exist.
- Every decision replays through input, versions, tools, memory, output, and rejection reason.
- Six gates and proposal→projected→sent→accepted/executed lineage remain distinct.
- Report improved and degraded axes, counterexamples, unmeasured values, and exact rollback tuple.
- If physical evidence is needed, stop with a request for a separate human approval card.

Safety
- Model outputs are human-reviewable candidates, not execution authority.
- Automatic recovery is limited to approved reversible skills and exposure budgets.
- Protective-stop/E-stop clearing and final approval remain with named people and devices.

Manufacturing Cell Checkpoint

Task schema: Are goal, prohibitions, completion, initial distribution, retry budget, and irreversible point versioned? Does ambiguity enter a clarification state? Data/logging: Are event, arrival, and use times plus plan, tool, memory, and call lineage retained? Metrics: Are precondition violations, rejection, latency, recovery, intervention, and force events compared to a baseline on matched denominators? Safety: Is the agent identity unable to alter controller, collision/force limits, watchdog, protective stop, or E-stop? Ownership: Are the named owners of planning, skill registry, evaluator, code promotion, physical approval, and rollback distinct?

If any answer is missing, keep physical execution closed. A success video, natural-language explanation, model confidence, or formally valid plan cannot replace human approval or a physical safety receipt. If an inherited S11 cell identity or S12 skill/control/evaluator version is missing, not measured, not exercised, or incompatible, the agent must not guess.

Relation to Prior Surveys

This chapter does not reteach the safe cell from S11 or the hybrid classical/learned execution spine from S12. It invokes those contracts as the agent’s lower boundary. Reader context is available in Coding Agents for Manipulation and RoboClaw and Agentic Long-Horizon Tasks. These links are contextual reading, not evidence for the claims in this chapter.

13. Turn counterexamples into the Chapter 8 operations contract

High task success is weak generalization evidence if evaluation items entered training, memory, or retrieval. When comparing a remembering system with a memory-free baseline, record evaluation exposure on the same axis. Remote model or tool updates can alter replay and operational latency. Human-assisted success must not be merged into autonomous success.

The interface to Chapter 8 is not a free-form conversation log. Hand over agent_version, authority table, skill/tool allowlists, planner/memory/retrieval/code versions, task/robot/data exposure, generalization axes, plan/call/rejection/recovery denominators, latency distribution, unsafe proposals, human interventions, force and stop events, contamination audit, failure bundle, and rollback tuple. An evaluator independent of the planner must check this record against sensor evidence and a fixed success definition.

What to Learn Next

Chapter 8 separates “a working demonstration” from trustworthy generality. This chapter’s authority table, skill-call schema, and failure/recovery state machine become evaluation targets. The next chapter separates within-task variation, new objects, scenes and instructions, new embodiment, and long horizon, then records contamination, intervention, latency, recovery, safety, and operating cost on an independent evaluation card.

Supplementary evidence map

The following verified primary-source groups come from the independent S13 ledger. They document search coverage without changing existing claims or denominators.

• demonstration and augmentation: ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback, DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignment, Semantically Controllable Augmentations for Generalizable Robot Learning, Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching — source-specific tasks, data, and evidence stages remain separate.
• uncertainty and recovery: Interactive Multi-Robot Flocking with Gesture Responsiveness and Musical Accompaniment, Partially Observable Task and Motion Planning with Uncertainty and Risk Awareness, RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning — source-specific tasks, data, and evidence stages remain separate.
• long-horizon skill composition: Local Policies Enable Zero-shot Long-horizon Manipulation, Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy, SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment — source-specific tasks, data, and evidence stages remain separate.

References

1. Brooks, R. A. (1986). A Robust Layered Control System for a Mobile Robot. IEEE Journal on Robotics and Automation. DOI:10.1109/JRA.1986.1087032.
2. Kaelbling, L. P., & Lozano-Pérez, T. (2011). Hierarchical Task and Motion Planning in the Now. IEEE ICRA. arXiv:1011.0010.
3. Paxton, C., Hundt, A., Jonathan, F., Guerin, K., & Hager, G. D. (2017). CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision. IEEE ICRA. arXiv:1611.06145.
4. Colledanchise, M., & Ögren, P. (2018). Behavior Trees in Robotics and AI: An Introduction. CRC Press. arXiv:1709.00084.
5. Garrett, C. R., Lozano-Pérez, T., & Kaelbling, L. P. (2018). FFRob: Leveraging Symbolic Planning for Efficient Task and Motion Planning. International Journal of Robotics Research. arXiv:1608.01335.
6. Garrett, C. R., et al. (2021). Integrated Task and Motion Planning. Annual Review of Control, Robotics, and Autonomous Systems. arXiv:2010.01083.
7. Mehmood, U., Sheikhi, S., Bak, S., Smolka, S. A., & Stoller, S. D. (2022). The Black-Box Simplex Architecture for Runtime Assurance of Autonomous CPS. NASA Formal Methods. DOI: 10.1007/978-3-031-06773-0_12. arXiv:2102.12981.
8. Ahn, M., et al. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. CoRL. arXiv:2204.01691.
9. Huang, W., et al. (2022). Inner Monologue: Embodied Reasoning through Planning with Language Models. CoRL. arXiv:2207.05608.
10. Raman, S. S., et al. (2022). ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. arXiv:2209.11302.
11. Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS. arXiv:2302.04761.
12. Liang, J., et al. (2023). Code as Policies: Language Model Programs for Embodied Control. IEEE ICRA. arXiv:2209.07753.
13. Chen, Y., Ding, Y., Branicky, M. S., & Zhang, Z. (2023). AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers. IEEE ICRA / arXiv. arXiv:2306.06531.
14. Ding, Y., Zhang, X., Paxton, C., & Zhang, S. (2023). Task and Motion Planning with Large Language Models for Object Rearrangement. IEEE/RSJ IROS. arXiv:2303.06247.
15. Liu, Z., Majumdar, A., Savarese, S., & Fragkiadaki, K. (2023). REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. CoRL. arXiv:2306.15724.
16. Rana, K., et al. (2023). SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. CoRL. arXiv:2307.06135.
17. Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR. arXiv:2210.03629.
18. Ahn, M., et al. (2024). AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents. arXiv:2401.12963v2.
19. Shah, R., Yu, A., Zhu, Y., Zhu, Y., & Martín-Martín, R. (2024). BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulation. arXiv:2410.06237v1.
20. Shaji, S., Huppertz, F., Mitrevski, A., & Houben, S. (2026). From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?. arXiv:2603.03148v1.
21. Chen, S., et al. (2024). ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback. arXiv primary preprint (2024-10-11). arXiv:2410.08464.
22. Chen, W., et al. (2024). DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignment. arXiv primary preprint (2024-10-15). arXiv:2410.11584.
23. Chen, Z., et al. (2024). Semantically Controllable Augmentations for Generalizable Robot Learning. arXiv primary preprint (2024-09-02). arXiv:2409.00951.
24. Chisari, E., et al. (2024). Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching. arXiv primary preprint (2024-09-11). arXiv:2409.07343.
25. Cuan, C., et al. (2024). Interactive Multi-Robot Flocking with Gesture Responsiveness and Musical Accompaniment. arXiv primary preprint (2024-03-30). arXiv:2404.00442.
26. Curtis, A., et al. (2024). Partially Observable Task and Motion Planning with Uncertainty and Risk Awareness. arXiv primary preprint (2024-03-15). arXiv:2403.10454.
27. Dai, Y., et al. (2024). RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning. arXiv primary preprint (2024-09-23). arXiv:2409.14674.
28. Dalal, M., et al. (2024). Local Policies Enable Zero-shot Long-horizon Manipulation. arXiv primary preprint (2024-10-29). arXiv:2410.22332.
29. Garcia, R., et al. (2024). Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy. arXiv primary preprint (2024-10-02). arXiv:2410.01345.
30. Garrett, C., et al. (2024). SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment. 2024 Conference on Robot Learning (CoRL). arXiv:2410.18907.