Skip to main content

Language Models + Robotics: The New Integration Layer for Humanoids

1. Why Humanoid Robots Need Language Models (LLMs + VLMs)​

... The quest to build truly autonomous and versatile humanoid robots has long been a grand challenge in artificial intelligence and robotics. While significant progress has been made in areas like locomotion, manipulation, and perception, a critical gap remains in enabling these robots to understand, interact with, and operate within complex human environments in a natural and intuitive manner. This is where large language models (LLMs) and vision-language models (VLMs) emerge as transformative integration layers.

Traditional robotic systems often rely on pre-programmed behaviors, explicit symbolic representations, or task-specific models trained on vast amounts of structured data. This approach, while effective for well-defined, repetitive tasks in controlled environments, rapidly breaks down when confronted with the inherent ambiguity, variability, and open-ended nature of the real world. Human environments are characterized by:

  • Semantic Richness: Objects have names, functions, and relationships (e.g., "cup on the table," "tool for tightening screws").
  • Contextual Nuance: Instructions are often vague and rely on common sense (e.g., "tidy up the room," "prepare a snack").
  • Dynamic Interactions: Humans communicate intent, provide feedback, and adapt goals mid-task.
  • Novelty and Generalization: Robots must handle unforeseen situations and extrapolate from limited experiences.

Language models, particularly those with billions of parameters trained on diverse internet-scale datasets, possess an unprecedented ability to:

  • Comprehend Natural Language: Parse and interpret human instructions, questions, and commands, extracting intent and relevant entities.
  • Leverage World Knowledge: Access and reason with a vast repository of common sense, factual information, and procedural knowledge embedded in their training data. This includes understanding object properties, typical human activities, and the physics of the world at a high level.
  • Perform Abstract Reasoning: Translate high-level goals into a sequence of actionable sub-goals, considering preconditions and effects.
  • Generate Explanations and Feedback: Communicate their understanding, ask clarifying questions, and report on task progress or failures in human-understandable terms.

Vision-language models extend these capabilities by bridging the modality gap between textual instructions and visual perception. VLMs can:

  • Ground Language in Perception: Associate linguistic concepts (e.g., "red mug," "screwdriver") with specific visual features in the robot's sensory input. This allows robots to identify and locate objects described by humans.
  • Understand Scene Semantics: Interpret the meaning of a visual scene beyond raw pixel data, recognizing objects, their attributes, and spatial relationships in a semantically meaningful way.
  • Perform Visual Question Answering (VQA): Answer questions about their visual environment using natural language, enabling more intuitive diagnostic and query capabilities.

For humanoid robots, the integration of LLMs and VLMs addresses several fundamental limitations:

  • Intuitive Human-Robot Interaction: Moves beyond rigid command interfaces to natural language dialogues, making robots accessible to a broader range of users. A user can simply say, "Please clean the living room," instead of programming each vacuuming motion.
  • Enhanced Task Understanding and Planning: LLMs can decompose complex, long-horizon tasks into manageable sub-goals, filling in implicit steps that a human might omit. For example, "make coffee" implies fetching water, coffee grounds, a filter, brewing, and pouring.
  • Robustness to Ambiguity and Novelty: When faced with an unfamiliar object or an ambiguous instruction, LLMs can leverage their vast knowledge to infer meaning or ask for clarification, rather than failing outright.
  • Adaptability to Dynamic Environments: As environments change or unexpected events occur, LLMs can replan and adapt their actions based on real-time linguistic and visual cues.
  • Accelerated Development and Deployment: By reducing the need for extensive manual programming for each new task or environment, LLM/VLM integration can significantly speed up the development and deployment of humanoid robots across various applications, from elder care to industrial assistance.

In essence, LLMs and VLMs provide humanoid robots with a form of "common sense" and a powerful communication channel, enabling them to transition from mere automatons performing predefined sequences to intelligent, adaptive partners capable of understanding and executing human intentions in the messy, unstructured reality of our world. This represents a paradigm shift from traditional, rigid control architectures to more flexible, cognition-driven robotic systems.

2. How Natural Language Becomes Robot Actions (NL → Intent → Skills → Control)​

The process of translating a human's natural language instruction into a robot's physical actions is a complex, multi-stage pipeline that requires sophisticated integration of language understanding, planning, and control systems. This transformation can generally be broken down into a hierarchical framework: Natural Language (NL) → Intent → Skills → Control.

2.1 Natural Language (NL) Input​

The starting point is unstructured natural language input from a human operator. This can take various forms:

  • Direct Commands: "Pick up the red ball and put it in the basket."
  • High-Level Goals: "Clean the kitchen."
  • Questions: "Where is my screwdriver?" (leading to an action to locate it).
  • Feedback/Correction: "No, not that one, the other one."

LLMs, often augmented with VLMs for visual context, are instrumental at this stage. They parse the raw linguistic input, identify key entities (objects, locations), verbs (actions), and adverbs/adjectives (modifiers), and extract the core communicative intent. This involves:

  • Syntactic and Semantic Parsing: Understanding the grammatical structure and meaning of the sentence.
  • Named Entity Recognition (NER): Identifying specific objects, persons, or locations.
  • Coreference Resolution: Linking pronouns and other anaphoric expressions to their referents (e.g., "it" refers to "the red ball").
  • Sentiment Analysis and Tone Detection: Inferring emotional context, which can be crucial for human-robot collaboration.

2.2 Intent Extraction and Representation​

Once the NL is parsed, the system must extract a formal, machine-readable intent. An intent is a structured representation of the user's desired goal or request, abstracting away the linguistic variability of the input. It defines what the human wants the robot to achieve, rather than how.

Intent representations can vary but typically include:

  • Action Verbs: A canonical set of robot-executable actions (e.g., pick, place, move_to, open, close).
  • Arguments/Parameters: Objects, locations, attributes, and other relevant contextual information (e.g., object: red_ball, destination: basket, location_type: shelf).
  • Constraints/Conditions: Specific requirements or safety parameters (e.g., speed: slow, grip_strength: gentle).

LLMs excel at this translation. Given a prompt that includes the NL input and a desired output format (e.g., JSON, YAML, or a domain-specific language for robot tasks), the LLM can generate the structured intent. For example:

{
  "action": "pick_and_place",
  "parameters": {
    "object": {"type": "ball", "color": "red"},
    "target_location": {"type": "basket"}
  },
  "constraints": []
}

This intent serves as the interface between the high-level cognitive understanding layer and the robot's action planning system.

2.3 Skill Selection and Planning​

Robots do not execute raw intents directly; they execute skills. A skill is a parameterized, reusable, and often learned behavior primitive that achieves a specific objective. Skills encapsulate the necessary perception, planning, and low-level control logic for a particular robot capability. Examples include:

  • Manipulation Skills: grasp_object(object_id, grip_force), place_object(pose, orientation).
  • Navigation Skills: navigate_to_pose(target_pose), follow_path(path_segments).
  • Interaction Skills: open_door(door_handle_id), push_button(button_id).

Given an extracted intent, the robot's skill manager or high-level planner uses the LLM's reasoning capabilities to:

  1. Select Appropriate Skills: Map the abstract intent to one or more relevant skills. An intent like pick_and_place might map to grasp_object followed by place_object.
  2. Decompose Complex Intents: For high-level goals like "clean the kitchen," the LLM can generate a sequence of intents that can be executed sequentially by available skills (e.g., clear_countertop → wipe_surface → empty_dishwasher). This involves leveraging common-sense knowledge about kitchen cleaning.
  3. Ground Skill Parameters: Populate the parameters of the selected skills with concrete values derived from the intent and the current environmental state (e.g., object_id from VLM detection, target_pose from semantic map).
  4. Handle Preconditions and Postconditions: Ensure that the environment is in a suitable state for a skill to execute (preconditions) and update the internal world model after skill execution (postconditions).

LLMs can act as powerful task planners or orchestrators at this level, generating a symbolic plan (e.g., a sequence of skill calls with grounded parameters) that can be executed by the robotic system. This plan can also include conditional branches, loops, and error recovery strategies generated by the LLM.

2.4 Low-Level Control and Execution​

The final stage involves the robot's control system executing the selected and parameterized skills. This is the domain of traditional robotics, involving:

  • Motion Planning: Generating smooth, collision-free trajectories for robot manipulators and mobile bases.
  • Inverse Kinematics/Dynamics: Calculating the joint angles and torques required to achieve desired end-effector poses or forces.
  • Motor Control: Sending commands to actuators (motors, servos) to execute the planned trajectories.
  • Sensor Feedback Integration: Using data from cameras, lidar, force/torque sensors, etc., to close control loops, ensure successful execution, and detect deviations.

Skills abstract away this low-level complexity. For instance, a grasp_object skill might internally involve:

  1. Perception: Using a VLM to detect and localize the object_id in 3D space.
  2. Pre-grasp Planning: Moving the robot arm to a safe pre-grasp pose.
  3. Grasp Planning: Computing a stable grasp pose on the object.
  4. Trajectory Generation: Planning the actual grasping motion.
  5. Force Control: Applying appropriate grip force based on object properties.
  6. Post-grasp Verification: Confirming the object is securely held.

While LLMs operate at the higher cognitive levels (NL → Intent → Skills), they do not directly control the robot's motors. Their role is to provide the intelligent, context-aware instructions that guide the robot's underlying control architecture. The modularity of this pipeline allows for the seamless integration of advanced AI reasoning with robust, real-time robotic control systems.

3. Vision-Language-Action (VLA) Pipelines with Concrete Examples​

Vision-Language-Action (VLA) pipelines represent a paradigm shift in robotics, enabling robots to interpret complex human instructions grounded in visual perception and translate them into a sequence of physical actions. At its core, a VLA system aims to bridge the gap between high-level human intent (expressed in natural language) and low-level robot control (executed in the physical world), all while continuously perceiving and understanding the environment.

3.1 Core Components of a VLA Pipeline​

A typical VLA pipeline integrates several key AI and robotics components:

  1. Vision-Language Model (VLM): The front-end of the pipeline, responsible for multimodal understanding. It takes raw visual input (e.g., camera images, depth maps) and natural language queries/commands, and produces a semantic understanding of the scene or identifies relevant objects/regions based on the linguistic prompt.
  2. Language Model (LLM) for High-Level Reasoning and Planning: This component receives the semantic understanding from the VLM and the initial natural language instruction. It is responsible for:
    • Intent Parsing: Converting NL instructions into structured, robot-executable intents.
    • Task Decomposition: Breaking down complex tasks into a sequence of simpler sub-tasks.
    • Action Planning: Generating a symbolic plan, often as a sequence of high-level skills, considering environmental states and robot capabilities.
    • Error Handling/Replanning: Detecting failures or unexpected situations and modifying the plan accordingly.
  3. Skill Library/Controller: A repository of pre-defined, robust robot skills (e.g., grasp, move_to, push, open). These skills abstract away the complexities of low-level motion planning and control.
  4. World Model/State Estimator: Maintains an up-to-date representation of the robot's environment, including object locations, properties, and the robot's own pose. This model is continuously updated by perceptual inputs and action outcomes.
  5. Robot Control System: Executes the low-level commands generated by the skill controller, interfacing directly with the robot's actuators and sensors.

3.2 Example VLA Pipeline: "Pick up the blue cube from the table and place it in the red bin."​

Let's walk through a concrete example to illustrate the flow:

Human Instruction: "Pick up the blue cube from the table and place it in the red bin."

Stage 1: Multimodal Perception and Grounding (VLM)

  • Input: Robot's camera feed (RGB-D images) + "blue cube," "table," "red bin."
  • VLM Processing:
    • The VLM analyzes the camera feed to identify objects within the scene.
    • It localizes and segments objects corresponding to "blue cube," "table," and "red bin." This involves object detection, semantic segmentation, and potentially 3D pose estimation.
    • Crucially, the VLM grounds these linguistic descriptions to specific pixel regions or 3D coordinates in the robot's perception space. For example, it might output (object_id: 001, label: 'blue cube', pose: P_cube), (object_id: 002, label: 'table', pose: P_table), (object_id: 003, label: 'red bin', pose: P_bin).
  • Output to LLM: A structured representation of the grounded entities and their poses.

Stage 2: High-Level Planning and Task Decomposition (LLM)

  • Input: Original NL instruction + Grounded entities from VLM.
  • LLM Processing:
    • Intent Parsing: The LLM interprets "Pick up X and place it in Y" as a PickAndPlace intent.
    • Task Decomposition: It breaks PickAndPlace into sub-tasks: NavigateToObject(blue_cube) → GraspObject(blue_cube) → NavigateToLocation(red_bin) → ReleaseObject(red_bin).
    • Parameter Grounding: It fills in the parameters for these sub-tasks using the VLM's grounded output: GraspObject(object_id=001, pose=P_cube), NavigateToLocation(pose=P_bin).
    • Precondition/Postcondition Checks: The LLM might consult a knowledge base to understand that GraspObject requires the gripper to be open (precondition) and results in the object being held (postcondition).
  • Output to Skill Controller: A symbolic plan or sequence of skill calls: 
    plan = [
        {"skill": "navigate_to_object", "params": {"object_id": "001", "pose": "P_cube"}},
        {"skill": "grasp_object", "params": {"object_id": "001", "pose": "P_cube"}},
        {"skill": "navigate_to_location", "params": {"target_pose": "P_bin"}},
        {"skill": "release_object", "params": {"target_location_id": "003", "target_pose": "P_bin"}}
    ]

Stage 3: Skill Execution and Low-Level Control

  • Skill Controller: Takes the plan and sequentially invokes the robot's skill library.
    1. navigate_to_object(object_id=001, pose=P_cube): The navigation skill plans a collision-free path to the blue cube's location and executes it via motor commands.
    2. grasp_object(object_id=001, pose=P_cube): The manipulation skill opens the gripper, moves the arm to the cube's pose, closes the gripper with appropriate force, and lifts the cube.
    3. navigate_to_location(target_pose=P_bin): The navigation skill moves the robot to the red bin.
    4. release_object(target_location_id=003, target_pose=P_bin): The manipulation skill moves the arm to a suitable release pose over the bin, opens the gripper, and retracts.
  • Feedback Loop: Throughout execution, sensor data (e.g., force sensors, camera) is continuously monitored. If a skill fails (e.g., grasp fails, collision detected), this information is fed back to the LLM for replanning.

3.3 Dynamic VLA Example: "Tidy up the desk."​

For more open-ended instructions, the VLA pipeline operates iteratively:

  1. Initial Query (VLM + LLM): VLM identifies all objects on the desk. LLM interprets "tidy up" as requiring objects to be categorized and moved to appropriate storage.
  2. Iterative Planning:
    • LLM asks: "What is this object?" pointing to a pen (via VLM bounding box).
    • Human: "That's a pen. Put it in the pen holder."
    • VLM grounds "pen holder." LLM generates PickAndPlace(pen, pen_holder).
  3. Execution & Loop: Robot executes the PickAndPlace skill. After completion, the VLM rescans the desk, and the LLM identifies the next untidy object, continuing the loop until the desk is deemed tidy (or until the LLM determines no more objects need tidying based on its world knowledge).

This iterative, perception-driven planning, enabled by the tight coupling of VLMs for grounding and LLMs for high-level reasoning, allows humanoid robots to tackle complex, underspecified tasks in dynamic, human-centric environments. The VLA pipeline transforms raw sensory data and natural language into purposeful, intelligent action.

4. ROS 2 + LLM Integration Architecture (Nodes, Services, Planners)​

ROS 2 (Robot Operating System 2) provides a flexible framework for building complex robotic applications through a distributed system of nodes, topics, services, and actions. Integrating Large Language Models (LLMs) into a ROS 2 architecture requires careful design to leverage the strengths of both systems: the LLM's high-level reasoning and the ROS 2's real-time communication and control capabilities. This section outlines common architectural patterns for such integration.

4.1 Foundational ROS 2 Concepts for LLM Integration​

Before delving into specific architectures, it's crucial to recall key ROS 2 concepts:

  • Nodes: Independent executable processes that perform computation (e.g., a camera driver node, a navigation node, a gripper control node).
  • Topics: Anonymous publish/subscribe mechanism for real-time data streaming (e.g., sensor data, robot state).
  • Services: Synchronous request/reply mechanism for remote procedure calls (e.g., request inverse kinematics, query object detection).
  • Actions: Long-running, goal-oriented tasks with feedback and cancellation capabilities (e.g., NavigateToPose action, PickAndPlace action).
  • Parameters: Dynamic configuration values for nodes.

4.2 Architectural Patterns for LLM Integration​

Integrating LLMs into ROS 2 typically involves creating dedicated ROS 2 nodes that encapsulate the LLM functionality and expose it to the broader robotic system via ROS 2 interfaces.

4.2.1 LLM as an Intent Parser (Service-Based)​

In this pattern, the LLM acts as a service that translates raw natural language commands into structured robot-executable intents. This is often the first layer of integration.

  • LLM Node: A ROS 2 node (llm_intent_parser_node) hosts the LLM (or interfaces with an external LLM API).
  • Service Interface: It exposes a ROS 2 service (e.g., /parse_command) that takes a std_msgs/String (the natural language command) as input.
  • Output: The service returns a custom ROS 2 message representing the structured intent (e.g., robot_msgs/Intent, containing fields like action_type, object_id, target_pose).
  • Example Flow:
    1. A human-robot interface (HRI) node publishes the command "Pick up the green bottle" to a /human_command topic.
    2. A coordinator node or task planner subscribes to /human_command and then calls the /parse_command service on llm_intent_parser_node.
    3. llm_intent_parser_node uses the LLM to generate Intent(action='grasp', object_name='green bottle', ...).
    4. The coordinator node receives this intent and proceeds to activate the appropriate skills.

Advantages: Clear separation of concerns, modular, allows for different LLMs to be swapped out easily. Disadvantages: Synchronous blocking nature of services might be less suitable for continuous, conversational interactions without careful design.

4.2.2 LLM as a High-Level Task Planner (Action-Based)​

For more complex, long-horizon tasks, the LLM can serve as a high-level planner, generating a sequence of actions or sub-goals for the robot. This typically involves a feedback loop.

  • LLM Planner Node: A ROS 2 node (llm_task_planner_node) acts as an action server (e.g., /plan_task).
  • Action Goal: The action goal could be a high-level NL command (e.g., "Make me coffee") or a structured intent from the previous stage.
  • Action Feedback: As the LLM generates sub-tasks or steps, it sends feedback to the action client. This could be the current sub-task, status updates, or requests for clarification.
  • Action Result: The final result is a complete sequence of executable skills/actions, or a report on task completion/failure.
  • LLM Role: The LLM might iteratively query the robot's world model (via ROS 2 services) for current state information (e.g., "What objects are on the counter?"), perform reasoning, and then output the next step of the plan.
  • Example Flow:
    1. A top-level HRI node sends an action goal to llm_task_planner_node with the command "Prepare breakfast."
    2. llm_task_planner_node uses the LLM to:
      • Query a semantic_map_node service (/query_objects) to see what food items are available.
      • Generate a plan: [get_cereal, get_milk, pour_cereal, pour_milk].
      • Send get_cereal as feedback.
    3. A skill_orchestrator_node receives get_cereal, then calls the grasp_action_server and navigate_action_server.
    4. Upon completion of get_cereal, the skill_orchestrator_node informs llm_task_planner_node, which then generates the next step (get_milk).

Advantages: Handles long-horizon tasks, provides inherent feedback for monitoring progress, supports dynamic replanning. Disadvantages: Requires more complex state management and robust error handling within the LLM interaction loop.

4.2.3 LLM as a World Model Query and Update Agent (Service/Topic-Based)​

LLMs can also be used to query and update a semantic world model based on perception and human input.

  • LLM World Node: A ROS 2 node (llm_world_model_node) processes queries and updates.
  • Services: Exposes services like /query_semantic_state (e.g., "Is the door open?") or /update_semantic_state (e.g., "The human just closed the cabinet").
  • Topics: May subscribe to vision topics (/perception/object_detections) to continuously update its internal representation of objects and their states, and publish simplified semantic updates to a /semantic_world topic for other nodes.
  • Example Use: A navigation planner could query /query_semantic_state before path planning to avoid moving through an "open door" that isn't visually perceived as such yet.

Advantages: Enriches the robot's understanding of its environment with semantic meaning and common sense. Facilitates natural language queries about the world.

4.3 ROS 2 Nodes for VLA Integration​

Based on these patterns, a full VLA pipeline in ROS 2 might involve a suite of interconnected nodes:

  • camera_driver_node: Publishes raw RGB-D images to /camera/image_raw, /camera/depth/image_raw.
  • vlm_perception_node: Subscribes to camera topics. Uses a VLM to perform object detection, segmentation, and pose estimation. Publishes perception_msgs/DetectedObjects and perception_msgs/GroundedEntities (linking NL labels to object IDs and poses) to /perception/grounded_objects.
  • llm_intent_parser_node: Provides the /parse_command service (as described above).
  • llm_task_planner_node: Provides the /plan_task action server (as described above). It orchestrates calls to llm_intent_parser_node, vlm_perception_node (via services for specific queries), and a skill_orchestrator_node.
  • skill_orchestrator_node: Subscribes to the planned skill sequence from llm_task_planner_node. Acts as a client for various robot-specific action servers (e.g., move_base_action_server, gripper_action_server, manipulation_action_server). It handles skill execution, monitors feedback, and reports status back to the llm_task_planner_node.
  • semantic_map_node: Maintains a persistent semantic map of the environment, updated by vlm_perception_node and potentially llm_world_model_node. Exposes query services.
  • human_interface_node: Manages user input (voice, text, gestures) and displays robot feedback. Interfaces with llm_intent_parser_node and llm_task_planner_node.

4.4 Planners in the ROS 2 + LLM Context​

Traditional ROS navigation and manipulation stacks already include sophisticated planners (e.g., Nav2 for navigation, MoveIt 2 for manipulation). The LLM primarily acts as a high-level symbolic planner that orchestrates these lower-level geometric and kinematic planners.

  • LLM as a Meta-Planner: Instead of planning joint trajectories, the LLM plans a sequence of calls to existing ROS 2 action servers (e.g., NavigateToPose, PickAndPlace). It provides the what and where, and the specialized ROS 2 planners handle the how.
  • Hybrid Planning: The LLM can generate symbolic sub-goals, which are then passed to a traditional symbolic planner (like a PDDL-based planner) to generate a more detailed, executable sequence. The LLM then acts as a PDDL domain/problem generator or a heuristic search guide.
  • Learning from LLM Plans: The sequences of actions generated by LLMs can also be used as training data for reinforcement learning (RL) agents or imitation learning to learn more efficient or specialized policies over time.

In conclusion, ROS 2 provides a robust, modular foundation for integrating LLMs and VLMs into robotic systems. By carefully designing ROS 2 nodes, services, and actions that encapsulate the LLM's reasoning capabilities, we can build sophisticated VLA pipelines that enable humanoid robots to understand, plan, and act intelligently in human-centric environments.

5. Grounding Language into Robot Perception (Scene Graphs, Semantic Mapping)​

For a humanoid robot to effectively execute natural language commands, it must accurately ground linguistic concepts into its physical perception and internal representation of the environment. This means associating words and phrases (e.g., "red mug," "on the table," "near the window") with specific objects, regions, and spatial relationships detected by its sensors. Without robust grounding, an LLM's abstract reasoning remains detached from the robot's operational reality. Key techniques for achieving this include scene graphs and semantic mapping.

5.1 The Challenge of Grounding​

The real world is inherently continuous, noisy, and high-dimensional, while language operates on discrete, symbolic, and often abstract concepts. Bridging this gap involves:

  • Object Recognition and Localization: Identifying specific instances of objects mentioned in language within the robot's sensor data (e.g., finding the blue cube amidst other objects).
  • Attribute Recognition: Detecting properties like color, size, material, and state (e.g., "open," "closed," "empty," "full").
  • Spatial Relation Understanding: Interpreting prepositions and spatial terms (e.g., "on," "under," "next to," "behind") in a geometrically meaningful way.
  • Abstract Concept Grounding: Relating high-level concepts like "tidy," "clean," or "safe" to quantifiable perceptual features and actions.

Vision-Language Models (VLMs) play a foundational role by directly learning these cross-modal associations from large datasets of image-text pairs. They can generate embeddings that represent both visual and linguistic inputs in a shared semantic space, enabling tasks like referring expression comprehension (locating an object described by text) and visual question answering.

5.2 Scene Graphs for Structured Perception​

A scene graph is a structured, symbolic representation of a visual scene that captures objects, their attributes, and their relationships. It converts raw pixel data into a graph where:

  • Nodes represent objects (e.g., chair, table, cup) and sometimes regions.
  • Edges represent relationships between objects (e.g., chair --[on]--> floor, cup --[on]--> table).
  • Attributes describe properties of objects (e.g., cup(color: red, material: ceramic)).

How LLMs/VLMs interact with Scene Graphs:

  1. VLM for Scene Graph Generation: A VLM, or a specialized perception module, processes the robot's visual input to detect objects, segment them, estimate their 3D poses, and infer spatial and functional relationships. The output is a raw scene graph.
  2. LLM for Scene Graph Refinement and Query:
    • Populating Attributes: The LLM can infer additional semantic attributes or relationships based on common sense knowledge not immediately apparent from raw perception (e.g., inferring chair --[supports]--> human if a human is detected sitting). It can also resolve ambiguities.
    • Language-to-Graph Query: When a human asks, "Where is the book on the desk?", the LLM translates this query into a graph traversal operation. It queries the scene graph for object(type: book, relationship: on, target: object(type: desk)). The VLM can then highlight the corresponding object in the robot's view.
    • Graph-to-Language Generation: The LLM can generate natural language descriptions of the scene based on the scene graph, allowing the robot to communicate its understanding (e.g., "I see a blue cup on the left side of the table.").

Benefits for Robotics: Scene graphs provide a compact, interpretable, and queryable representation of the environment that aligns well with the symbolic reasoning capabilities of LLMs. They make it easier for LLMs to plan actions based on object relationships and to provide detailed feedback to humans.

5.3 Semantic Mapping for Spatial Grounding​

Semantic mapping extends traditional geometric mapping (e.g., SLAM - Simultaneous Localization and Mapping) by incorporating semantic information about objects and regions. Instead of just a grid of occupied/free space, a semantic map includes:

  • Object Classes: Differentiating between "chair," "table," "door," "wall."
  • Object Instances: Tracking specific instances of objects with unique IDs and their estimated poses over time.
  • Room/Area Labels: Labeling regions as "kitchen," "living room," "bedroom."
  • Navigable/Non-Navigable Areas: Encoding traversability information.
  • Affordances: Information about what actions can be performed with/on objects (e.g., a "door" can be opened or closed).

How LLMs/VLMs interact with Semantic Maps:

  1. VLM for Semantic Annotation: VLMs are crucial for populating semantic maps by identifying objects, segmenting regions, and classifying them based on visual features. This allows the map to be built incrementally as the robot explores.
  2. LLM for Map Query and Planning:
    • High-Level Navigation: When a human says, "Go to the kitchen and find the milk," the LLM can query the semantic map for the kitchen region, then, once there, query for milk objects within that region. The navigation stack then uses the geometric part of the map for path planning.
    • Contextual Understanding: The LLM can use the semantic map to understand spatial context (e.g., "the object in the pantry" vs. "the object on the counter").
    • Goal Representation: Goals can be defined semantically (e.g., reach(object: milk, location: kitchen)), which the LLM translates into a sequence of navigation and manipulation skills grounded in the semantic map.

Benefits for Robotics: Semantic maps enable robots to understand and execute commands that involve high-level locations and object categories, providing a more human-like spatial reasoning capability. They are essential for long-horizon navigation and manipulation tasks in complex, large-scale environments.

5.4 Integrated Grounding Framework​

An advanced grounding framework might combine both scene graphs and semantic maps:

  • Local Scene Graphs: For detailed interaction within the robot's immediate vicinity (e.g., manipulating objects on a desk).
  • Global Semantic Map: For large-scale navigation and understanding of room-level or building-level context.
  • LLM as Integrator: The LLM acts as the central reasoning engine, interpreting instructions, querying the appropriate representation (scene graph or semantic map) based on the scope of the command, and generating skill parameters grounded in the combined spatial and semantic knowledge.

This robust grounding mechanism, powered by VLMs for perception and LLMs for semantic reasoning, is critical for enabling humanoid robots to operate intelligently and autonomously in diverse human environments, transforming abstract language into concrete, actionable understanding of their surroundings.