Chapter 5: Ground Language and Scenes into Action — Open Vocabulary, Spatial Understanding, and Skill Selection

Overview

“Insert the blue bracket into the right hole” is not a robot command. It does not yet identify which blue object, which part of the bracket, whose viewpoint defines right, how insertion is judged, or whether the relevant skill preconditions hold. The output of language grounding must therefore be a typed task proposal with evidence, not motor values. reject and clarify are normal outputs whenever the request cannot be made sufficiently specific.

S11 contributes the physical cell’s cameras, clocks, frames, calibration, object assets, workspace, controllers, collision and force limits, watchdog, protective stop, E-stop, and accountable human. S12 contributes versioned task and skill APIs, preconditions, completion and failure conditions, feasibility projection, proposal → projected → sent → accepted/executed lineage, independent evaluation, and rollback. Chapter 4 fixed the adapted model’s camera, frame, action, and gripper contracts. This chapter asks a new question: how can language and scene evidence be converted into objects, parts, relations, goal states, and skill preconditions while ambiguous or physically unverifiable requests are closed before command issuance?

After reading this chapter, you will be able to... - convert an instruction into typed task constraints and goal states while preserving provenance, viewpoint, frame, and units; - choose among open-vocabulary recognition, 2D and 3D representations, affordances, scene graphs, and contact-feasible geometry on matched axes; - turn paraphrase, contradiction, negation, occlusion, missing prerequisites, forbidden requests, and missing units or frames into separate rejection tests; - build a language→task→skill contract that separates semantic scores, physical feasibility, and safety authority; and - diagnose failures in language, perception, state estimation, geometry, skills, control, and evaluation in causal order.

The experiment question is: with the tabletop-assembly initial distribution, cameras and calibration, skill library, lower planner and controller, evaluator, and safety limits fixed, does the system propose the correct goal state and skill under paraphrased instructions, new objects, layouts, and occlusions—and reproducibly reject ambiguity, contradiction, hidden prerequisites, and forbidden requests before issuing a real command?

1. Mental model: from words to a typed proposal, not torque

Language and a scene are not two guessers licensed to fill each other’s gaps. The instruction supplies requirements and prohibitions; the scene supplies currently observable entities and relations. Their combined contract is still not execution. It is a typed candidate that downstream components can inspect.

natural-language instruction + dialogue context
            ↓ parse scope, reference, negation, units, viewpoint
typed intent: object | part | relation | goal state | constraints
            ↓ bind to current scene evidence and instance IDs
2D candidates → 3D state → scene relations → affordance hypotheses
            ↓ ambiguity, contradiction, occlusion, precondition gates
versioned task proposal or reject/clarify
            ↓ match allowlisted skills and their preconditions
versioned skill-call candidate
            ↓ TAMP, IK, collision, contact geometry, force, deadline checks
projected proposal → sent → accepted/executed

Outside every path: independent evaluator · collision/force gates · watchdog ·
protective stop · E-stop · human approval

Task and motion planning (TAMP) is a bridge that jointly considers symbolic goals and continuous geometry. A planner can only inspect the objects, action effects, and geometry represented in its model, however. Undetected obstacles, stale scenes, real friction, tolerance, deformation, and contact force remain responsibilities of current observation and downstream execution gates.

In SayCan's 101-instruction evaluation, planning success was 84% and execution success was 74%, separating fluent planning from executable skills. ^[8] The experiment uses a real mobile manipulator in kitchen and office settings; that source task, robot, denominator, and evidence stage must remain attached. The system depends on a human-defined/trained skill library and value functions, and the multiplied score is not a calibrated safety guarantee. The result must not be imported as tabletop-insertion success or universal language understanding.

Reflect that gap in the schema rather than one undifferentiated “success” field. Record intent_parse_pass, entity_grounding_pass, goal_match, skill_preconditions_pass, geometry_pass, and execution_success separately. Otherwise a correct paraphrase can hide a wrong object, and successful low-level recovery can be miscredited to the language layer.

2. The language→task→skill contract: turn one instruction into an auditable record

Use this through-task instruction:

“From the robot-base viewpoint, grasp the blue slotted bracket on the left, stop 15 mm above the marked socket A, and insert it. Do not touch the red cap.”

Do not translate it directly to pick_and_insert(). Attach evidence, versions, and failure states to every row of the contract first.

The goal must be an evaluable state, not a verb. “Insert” is underspecified; axis error ≤ task tolerance, depth ∈ accepted interval, no contact with forbidden object, no force event, and independent evaluator passed can be tested. The language model does not invent tolerances. They come from the versioned task, part drawings, and validation record. A missing number or unit remains missing and closes the gate.

A skill is not grounded by name alone either. Each call declares its input frame, units, deadline, observation freshness, preconditions, completion and failure states, fallback, and authorized caller. The language layer may propose a skill. It cannot change an API version, create a non-allowlisted skill, or relabel a failed postcondition as success.

3. Choose scene representations on matched axes

Open vocabulary means querying beyond a closed training-time class list. It is not a single property called “understanding a new object.” Boxes, masks, depth, pose, relations, action-effect evidence, and contact geometry answer different questions.

CLIP supports transferable visual representation learned from natural-language supervision ^[6], but semantic similarity does not thereby include assembly geometry or contact semantics. OWL-ViT and Grounding DINO study text-conditioned 2D detection, while Segment Anything studies promptable segmentation ^{[9] [13] [15]}. These outputs can be composed, but uncertainty and instance identity must survive every transformation.

Open-vocabulary boxes and masks are not object pose, collision geometry, grasp stability, or contact feasibility. ^{[9] [13] [15]} The load-bearing evidence in those sources is from offline image detection and segmentation benchmarks, not assembly execution. A high detection score therefore remains a perception_candidate until depth, part, pose, occupancy, calibration, and contact tolerance are checked.

There need not be one winner in a “2D versus 3D” contest. Use 2D to connect a query to pixels, bind candidates across views and depth to an instance, transform the instance into the work frame, let a scene graph preserve reference relations, and let a planner inspect continuous geometry. Hidden shape is an estimate, not an observation, and must be labeled accordingly.

4. Affordance and spatial understanding: bind relations to action effects

The formalization by Şahin et al. treats an affordance as an entity–behavior–effect relation, not an object label ^[1]. The same cylinder may or may not afford grasping depending on parallel-gripper width, approach direction, surface, and current occupancy. Object-action complexes provide a related probabilistic view that learns actions and outcomes together ^[2]. Neither taxonomy supplies a calibrated detector or a contact validator by itself.

CLIPort combines a semantic “what” path with a spatial “where” path for language-conditioned planar pick-and-place, while SpatialVLA introduces egocentric 3D positional encoding and adaptive action grids into a VLA ^{[10] [18]}. SayPlan uses 3D scene graphs as structured spatial context for task planning over large environments ^[17]. This lineage supports better connections among meaning, relations, and action candidates. Its success numbers cannot be pooled because robot, data, action space, and evaluation procedures differ.

Affordance and language-conditioned evidence binds entities and relations but curated or planar scenes do not establish occluded insertion feasibility. ^{[1] [10] [18]} The evidence can also associate expected effects, but CLIPort’s ten language-conditioned tabletop tasks and real-robot demonstrations, and SpatialVLA’s spatial-representation evaluation, retain their own denominators. Representation gains are task-specific; hidden surfaces, socket occupancy, tolerance, friction, and insertion force require separate 3D observation, a pre-contact stop, and lower-level force evaluation.

A scene graph is not a truth ledger. Attach source_sensor, event_time, frame_id, calibration_generation, confidence, visibility, and last_verified to every node and edge. left_of(part-17, socket-A) can reverse between camera and base viewpoints. inside(part-17, socket-A) cannot be established by 2D overlap. Evaluate relations as 3D predicates with an explicit frame and margin.

Contact-feasible geometry begins below semantics. A grasp candidate checks finger approach, closing volume, and center of mass. A transport candidate checks collision margin, fixtures, and cables. An insertion candidate includes a pre-contact pose, axis alignment, tolerance, and retreat path. Real contact still belongs to approved impedance or force control and independent limits. The text token “insertable” carries no force authority.

5. Formal checking is not a live scene

Translating language into temporal logic or planning predicates enables mechanical checks of contradiction and ordering. TAMP can alternate symbolic choices with geometry checks rather than committing to a complete but geometrically impossible plan ^[3]. AutoTAMP studies an architecture in which an LLM translates and critiques while formal constraints and motion feasibility are checked by a planner ^[16].

AutoTAMP separates language translation from checking and motion feasibility, but only encoded constraints can be checked. ^{[16] [3]} The cited evidence is primarily simulation and modeled planning. Stale perception, undetected obstacles, incorrect calibration, and unmodeled contact or friction remain outside the formula, so passing a formal check is not physical-execution certification.

Keep two checks separate:

1. Static contract check: required fields, types, units, frames, allowlisted skills, prerequisite/completion/prohibition predicates, deadlines, and caller authority.
2. Dynamic world check: current-sensor confirmation of identity, visibility, occupancy, pose, path, collision, pre-contact state, and force zero.

The static check is reproducible and fast but can become stale. The dynamic check sees the current world but inherits sensor and calibration failures. Preserving distinct components and reason codes reduces common-cause failure and makes remediation attributable.

6. Ambiguity is an evaluation denominator, not an exception

A benchmark of only satisfiable instructions rewards visual habit. If training always puts one object into one socket, a policy can score highly while ignoring language. Minimal pairs that hold the scene fixed and change one word, relation, or logical condition expose that behavior.

Grounding gates must test paraphrase, contradiction, negation, missing units/frames, occlusion, missing prerequisites, and forbidden requests separately. ^{[19] [20]} Attribute and relation contradictions need their own slices. ICBench diagnoses linguistic blindness under structured contradictory instructions; its main denominator is 30 LIBERO simulation tasks with 50 rollouts per task variant. The same paper separately demonstrates IGAR on one Franka Research 3 cube-to-drawer task, but reports no hardware trial count, and that single task does not validate the complete rejection matrix on physical hardware. Grounded Decoding also depends on its allowed action vocabulary and grounded scorer. No single calibrated method covers the complete matrix, so local rejection tests and their real failure denominators remain necessary.

6.1 Ambiguity and rejection tests for tabletop assembly

Do not collapse this table to one accuracy score. Rejecting a case that needs clarification is also a failure; responding to a forbidden request with a question is a failure of a different kind. Measure correct state transition, unsafe false acceptance, unnecessary rejection, post-question resolution, and time to human handoff per input class. High “task success” under contradictions is a warning to test whether language was ignored.

7. Minimal reproducible workflow and validation gates

G0 — Seal inherited contracts

Hash the S11 cell, sensor, frame, calibration, control, and safety identities; the S12 task, skill, projection, evaluation, and rollback versions; and Chapter 4’s adaptation tuple and camera/action semantics. Record absent information as missing, skipped fault tests as not exercised, absent measurement as not measured, and semantic mismatch as incompatible. Every such state closes the dependent gate.

G1 — Validate the language schema first

Preserve the original text and dialogue scope. Convert object, part, relation, goal, prohibition, unit, and frame into typed fields. Arbitrary free text cannot become a skill argument. Schema failure terminates in clarify or reject, not an unbounded model retry loop.

G2 — Bind scene evidence to an instance and a time

Connect 2D candidates, masks, depth, pose, and relations to one instance ID while preserving raw outputs and transformations. Store observation event, arrival, and use times plus calibration generation. Hidden state remains unknown; expired graphs are discarded.

G3 — Pass semantic minimal pairs

Paraphrases should produce the same contract; changes in attribute, relation, and negation should produce a different one. When instruction and scene conflict, do not select a visually familiar action. Replay A01–A12 deterministically for every model, prompt, and perception version.

G4 — Check task and skill preconditions

Verify that the goal is observable and consistent with prohibitions. Query only allowlisted skills and match their inputs, preconditions, postconditions, deadlines, and retreat behavior. Do not generate free-form code or a new API merely because no skill fits.

G5 — Check geometry and pre-contact feasibility

Run TAMP, IK, workspace, collision, speed, and pre-contact-pose checks. Insertion is a distinct stage from free-space transport. Without an approved contact skill and force limits, the state is incompatible. A semantic score cannot bypass this gate.

A grounded skill request still passes versioned preconditions, geometry, collision, force, deadline, and human approval. ^{[8] [16] [5]} SayCan’s real-robot evidence and AutoTAMP’s simulation evidence support separating high-level proposals from feasibility checks; Black-Box Simplex supports independent runtime assurance and controller fallback under its explicit dynamics and timing bounds. None certifies the complete contact-assembly stack, and semantic confidence cannot replace physical validators.

G6 — Promote through separated evidence stages

First test contract conversion and rejection on fixed recorded replay, then verify zero non-allowlisted calls with controller stubs. Test geometry, queue behavior, and injected faults in simulation. Use shadow execution to measure real observation age and proposal distributions. A hardware trial requires a separate human-reviewed bounded-trial card; the Codex task in this chapter does not execute it.

8. Do not call every failure a semantic-model failure

Use this diagnostic order: language scope → instance identity → part → time → frame and units → task logic → skill preconditions → geometry → contact → control and evaluation. If changing phrasing flips the outcome, inspect grounding. If changing camera or calibration generation flips it, inspect scene binding. If the same grounded contract is repeatedly rejected by the motion planner, inspect pose, occupancy, and workspace before editing the prompt.

8.1 Read evidence stages and counterevidence together

The table explains why cross-paper success rates should not be pooled. Robots, tasks, data, action spaces, reset and exclusion rules, and evidence stages differ. Labels such as “open vocabulary,” “spatial understanding,” and “real robot” do not create comparability. The local experiment compares outputs under one task version, one A01–A12 denominator, and fixed geometry, control, and safety contracts.

9. Metrics and artifacts to retain

The mandatory bundle is instruction_contract_schema, instruction_suite, entity_grounding_ledger, scene_graph_snapshot, affordance_hypotheses, grounded_task_contract, skill_selection_trace, ambiguity_rejection_report, proposal_execution_lineage, synchronized_trace, fault_injection, and grounding_receipt. Every record carries model, prompt, perception, scene, task, skill, calibration, and evaluator versions.

Terry’s ForceVLA explainer and DexForce explainer are companion readings on adding force information to action representations. They are reader crosslinks, not evidence for this chapter’s claims. A force-aware model still does not own force limits or stop authority.

9.1 Bounded Codex implementation prompt

Goal
- Evaluate a language→task→skill contract converter on recorded tabletop scenes and instructions.
- Run A01–A12 ambiguity, contradiction, negation, missing-state, and forbidden-request tests plus fault injection.
- Produce grounded proposal/reject/clarify records and an evaluation report, never execution commands.

Context
- Read S11 cell/sensor/clock/frame/calibration/safety identities.
- Read S12 task/skill/precondition/postcondition/projection/evaluator/rollback contracts.
- Hold the Ch4 camera/action/gripper/adaptation tuple fixed.
- Use only recorded images, depth/point clouds, scene snapshots, and controller stubs.

Constraints
- Do not call robot drivers, send_action, controller switching, force/limit change, or fault-clear APIs.
- Never pass free text as a skill argument; use a typed schema and allowlist.
- Preserve missing/not measured/not exercised/incompatible/unknown rather than guessing.
- Preserve observation event/arrival/use times and frame/unit/calibration generation.
- Do not merge proposal/projected/sent/accepted-executed IDs.
- Never bypass geometry/collision/force/deadline/human approval using semantic confidence.

Done when
- instruction_contract_schema and the language→task→skill table exist.
- Every instruction has evidence-linked object/part/relation/goal/constraint/precondition fields.
- A01–A12 have expected state, actual state, reason code, and deterministic replay hash.
- Open-vocabulary box/mask, 3D pose, scene relation, affordance, and contact geometry are scored separately.
- Report false accept, unnecessary reject, clarification resolution, stale rejection,
  skill-precondition violation, geometry rejection, latency, and human handoff.
- grounding_receipt identifies the exact prior tuple and a restore test.

Safety
- Do not execute a hardware trial or clear a stop.
- Outputs are bounded skill proposals or reject/clarify only.
- Protective stop, E-stop, and final approval remain with independent devices and humans.

10. Limits and the interface to Chapter 6

This evidence packet does not validate one complete assembly stack. Image benchmarks do not test robot contact; planar pick-and-place does not stand for occluded precision insertion; and formal planning cannot check an unencoded world. Recent spatial VLA and contradiction benchmarks are useful research leads, but local hardware, data overlap, long horizons, latency and queues, force events, and independent replication remain open. A contract that stops at unknown matters more than a larger model that fills the field by guessing.

Chapter 6 receives no text plan or executable trajectory. It receives a versioned goal state; evidence-linked objects, parts, relations, and observation times; allowlisted skill candidates; prerequisites, prohibitions, and completion conditions; geometry-check results; reject/clarify reasons; and evidence stages. A world model may predict candidate outcomes or failures and rank plans under that contract. It may not turn hidden state into fact, turn predicted success into execution approval, or inherit collision, force, and real-time authority.

Prediction should therefore be evaluated not by a plausible future video, but by whether it predicts this chapter’s goal predicates and failure classes, filters bad candidates before real execution, and stops when uncertain. Chapter 6 addresses precisely that gap from prediction → candidate requiring verification → execution approval.

Supplementary evidence map

The following verified primary-source groups come from the independent S13 ledger. They document search coverage without changing existing claims or denominators.

• language and spatial representations: Visual Language Maps for Robot Navigation, CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision, Explainable deep learning improves human mental models of self-driving cars, CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance — source-specific tasks, data, and evidence stages remain separate.
• action and affordance links: CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation, LLaRA: Supercharging Robot Learning Data for Vision-Language Policy, OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics — source-specific tasks, data, and evidence stages remain separate.
• planning and physical compliance: Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following, RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-World, Language Conditioned Multi-Finger Dexterous Manipulation Enabled by Physical Compliance and Switching of Controllers — source-specific tasks, data, and evidence stages remain separate.

References

1. Şahin, E., Çakmak, M., Doğar, M. R., Uğur, E., & Üçoluk, G. (2007). To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control. Adaptive Behavior.
2. Montesano, L., Lopes, M., Bernardino, A., & Santos-Victor, J. (2008). Learning Object-Action Complexes: A Probabilistic Generative Approach. IEEE Transactions on Robotics.
3. Kaelbling, L. P., & Lozano-Pérez, T. (2011). Hierarchical Task and Motion Planning in the Now. IEEE ICRA. arXiv:1011.0010.
4. Paxton, C., et al. (2017). CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision. IEEE ICRA. arXiv:1611.06145.
5. Mehmood, U., et al. (2022). The Black-Box Simplex Architecture for Runtime Assurance of Autonomous CPS. NASA Formal Methods. DOI: 10.1007/978-3-031-06773-0_12. arXiv:2102.12981.
6. Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML. arXiv:2103.00020.
7. Zeng, A., et al. (2021). Transporter Networks: Rearranging the Visual World for Robotic Manipulation. CoRL. arXiv:2010.14406.
8. Ahn, M., et al. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. CoRL. arXiv:2204.01691.
9. Minderer, M., et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV. arXiv:2205.06230.
10. Shridhar, M., Manuelli, L., & Fox, D. (2022a). CLIPort: What and Where Pathways for Robotic Manipulation. CoRL. arXiv:2109.12098.
11. Shridhar, M., Manuelli, L., & Fox, D. (2022b). Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation. CoRL. arXiv:2209.05451.
12. Raman, S. S., et al. (2022). ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. arXiv:2209.11302.
13. Liu, S., et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv:2303.05499.
14. Huang, W., et al. (2023). Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control. NeurIPS. arXiv:2303.05771.
15. Kirillov, A., et al. (2023). Segment Anything. ICCV. arXiv:2304.02643.
16. Chen, Y., Ding, Y., Branicky, M. S., & Zhang, Z. (2023). AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers. IEEE ICRA / arXiv. arXiv:2306.06531.
17. Rana, K., et al. (2023). SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. CoRL. arXiv:2307.06135.
18. Qu, D., et al. (2025). SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model. Robotics: Science and Systems. arXiv:2501.15830v5.
19. Zhang, N., Zhu, B., Zhou, S., & Chen, J. (2026). Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration. arXiv:2603.06001v2.
20. Huang, C., et al. (2023). Visual Language Maps for Robot Navigation. IEEE ICRA. arXiv:2210.05714.
21. Kang, G., et al. (2024). CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision. arXiv primary preprint (2024-11-01). arXiv:2411.00508.
22. Kenny, E. M., et al. (2024). Explainable deep learning improves human mental models of self-driving cars. arXiv primary preprint (2024-11-27). arXiv:2411.18714.
23. Li, J., et al. (2024). CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance. arXiv primary preprint (2024-12-29). arXiv:2412.20451.
24. Li, Q., et al. (2024). CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation. arXiv primary preprint (2024-11-29). arXiv:2411.19650.
25. Li, X., et al. (2024). LLaRA: Supercharging Robot Learning Data for Vision-Language Policy. arXiv primary preprint (2024-06-28). arXiv:2406.20095.
26. Liu, P., et al. (2024). OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics. arXiv primary preprint (2024-01-22). arXiv:2401.12202.
27. Ma, Y., et al. (2024). Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following. arXiv primary preprint (2024-08-02). arXiv:2408.01147.
28. Mao, W., et al. (2024). RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-World. arXiv primary preprint (2024-11-29). arXiv:2412.00171.
29. Pan, C., et al. (2024). Language Conditioned Multi-Finger Dexterous Manipulation Enabled by Physical Compliance and Switching of Controllers. arXiv primary preprint (2024-10-17). arXiv:2410.14022.