Building Vision-Language-Action Policies for Robotic Manipulation

Project status: ongoing. This article documents a controlled research progression for learning robotic manipulation policies from vision, language, and action data.

Why Vision-Language-Action Models Matter

A robot policy should not only see the world. It should also understand what task it is being asked to do, and then choose actions that move the task forward.

This project studies that idea in a small controlled manipulation setting. The environment is intentionally simple: a 2D workspace, colored objects, a gripper, and language instructions such as reach the red object. The simplicity is useful because it makes every failure visible. If the model moves toward the wrong object, ignores language, or cannot recover from a bad position, the problem can be inspected directly.

Custom Dataset and Task

I built the dataset as a synthetic 2D manipulation benchmark. Each scene contains a small set of colored objects placed at random positions on a square workspace. A gripper is also placed in the workspace, and one object color is selected as the target for that sample.

The language instruction describes the task in a short sentence, for example reach the red object. The image shows the full scene, the state records the current gripper position, and the training label comes from a simple expert controller that points from the gripper toward the instructed target.

instruction = f"reach the {target_color} object"
image = render_scene(object_map, gripper_xy, image_size=image_size)
target_xy = object_map[target_color]
action = oracle_action(gripper_xy, target_xy, max_step=max_step)

This gives each sample four connected parts: what the robot sees, what it is told to do, where it currently is, and what action would move it closer to completing the task. The benchmark is small, but it captures the core structure of visual-language-action learning for manipulation.

The project progresses through several methods. Each method changes one important part of the model or dataset while keeping the task understandable.

Direct Action Regression

Direct action regression is the simplest possible VLA policy. The model receives an image, a language instruction, and the gripper state. It directly predicts the next action.

instruction = f"reach the {target_color} object"
image = render_scene(object_map, gripper_xy, image_size=image_size)
action = oracle_action(gripper_xy, target_xy, max_step=max_step)

This method proves the basic pipeline: generate synthetic scenes, tokenize language, render images, train a CNN policy, and evaluate it in closed loop.

The limitation is that direct action regression asks the model to solve too many things at once. It must identify the correct object, estimate its position, compare it with the gripper state, and output the correct movement. The final validation action error was around 0.053, but closed-loop tests still failed. This showed that a low supervised loss is not enough for reliable policy behavior.

Target-First Learning

Target-first learning changes the prediction target. Instead of predicting the action directly, the model predicts where the instructed object is. The action is then computed analytically from the predicted target and the current gripper state.

This makes the problem easier to understand. The network is responsible for perception and grounding: where is the red object? The controller is responsible for movement: move from here toward that point.

target_heatmap = make_target_heatmap(target_xy, size=heatmap_size)
pred_target = soft_argmax(predicted_heatmap)
action = controller(pred_target - state)

This method also adds coordinate channels and noisy off-policy gripper states. These help the model reason about geometry and recovery situations. Validation action error improved to about 0.0278, and the target error became measurable directly.

FiLM Conditioning

FiLM conditioning keeps the target-first objective but changes how language enters the visual model. Instead of using language only near the end, it uses language-conditioned modulation inside the CNN.

FiLM means the language instruction produces scale and bias values that modulate visual feature maps. In simple terms, the phrase red object changes how the visual network processes the scene.

This is useful because language should shape perception early, not only after the image has already been compressed. The experiment tests whether stronger language-conditioned visual features improve object grounding.

Cross-Attention Fusion

Cross-attention fusion replaces FiLM with a small cross-attention mechanism. The CNN produces spatial visual tokens, and the language embedding attends over those tokens.

The motivation is different from FiLM conditioning. FiLM modulates channels globally. Cross-attention gives the instruction a more direct way to select spatial evidence. For example, the language token for red should attend more strongly to the region containing the red object.

The rest of the pipeline remains fixed: target heatmap prediction, soft argmax, and analytic control. This makes the comparison cleaner because the main change is the multimodal fusion mechanism.

Token Transformer Fusion

Token transformer fusion moves closer to modern VLA design. The visual feature map is converted into tokens, the instruction becomes a text token, and a small transformer mixes them together.

This allows repeated interaction between visual and language information. Instead of one attention step, the model can refine the relationship between task words and object locations across multiple transformer layers.

This method produced the strongest saved report among the early exported results. Validation action error reached about 0.0197, and validation target error reached about 0.095. However, closed-loop success was still limited: around 6.7% overall in the capability test, with harder starts failing more often.

That result is important. It shows that better prediction metrics do not automatically create robust robot behavior. Closed-loop evaluation is necessary.

Patch-Token Policy

Patch-token policy explores a token-native visual encoder. It removes the CNN feature hierarchy and patchifies the image directly. Each image patch becomes a token. Language and state are also represented as tokens, and a transformer mixes everything together.

This method asks a clean architectural question: can a token-native visual policy solve the same grounding task without relying on CNN features?

I have kept this section without a result GIF for now because an execution demo has not been added to the site assets yet. The comparison remains clear: keep the same target-first supervision and analytic controller, then isolate the effect of the visual representation.

Temporal Manipulation Policy

Temporal manipulation moves from reactive reaching to sequential manipulation. The instruction becomes more complex: pick the source object and place it on the target object.

The dataset stores short episodes instead of isolated frames. Each training sample contains recent images, robot states, previous actions, one language instruction, and a chunk of future actions.

sample = {
    "images": history_images,
    "states": history_states,
    "prev_actions": previous_actions,
    "instruction": instruction,
    "target_actions": future_action_chunk,
}

This is the most important conceptual shift in the project. Real manipulation is not a single-frame problem. The correct action depends on what happened before: whether the gripper approached the object, whether it picked it up, and whether it is now moving toward the target.

This section also stays without a result GIF for now because an execution demo has not been added to the site assets yet. Conceptually, temporal manipulation is the bridge from simple VLA grounding toward sequence-aware robot policy learning.

Discussion: What Changed Across the Methods

The method progression is useful because each step isolates a different part of the VLA problem.

Direct action regression treats manipulation as direct action prediction. This is the simplest formulation, but it hides too much inside one prediction head. The model must understand the image, ground the language, reason about geometry, and output the correct action all at once. The result is a useful baseline, but it is difficult to interpret when it fails.

Target-first learning makes the problem more structured by separating target localization from control. This is a major improvement in clarity. If the policy fails, I can inspect whether the target estimate is wrong or whether the controller is behaving poorly. This method also shows why intermediate representations matter in robotics: predicting a target heatmap is more interpretable than predicting an action vector directly.

FiLM conditioning and cross-attention fusion focus on how language interacts with vision. FiLM lets the instruction shape visual feature extraction through channel-wise modulation. Cross-attention makes the interaction more spatial by letting language attend over visual regions. Both methods are attempts to answer the same question: how should the phrase red object influence what the model sees?

Token transformer fusion moves toward a more modern multimodal structure by mixing visual tokens and a language token inside a transformer. This gives the model more opportunities to refine the relationship between instruction and scene. It also produced the best saved supervised metrics among the exported runs. However, the low closed-loop success rate shows that better one-step prediction does not automatically produce reliable behavior over time.

Patch-token policy and temporal manipulation point toward the next research direction. Patch-token policy changes the visual representation by using image patches directly. Temporal manipulation changes the task structure by moving from single-step reaching to action history and task progress. Together, they mark the shift from small visual grounding experiments toward policies that must reason over time.

Overall, the strongest improvement is not one specific architecture. It is the gradual separation of the problem into parts that can be measured: grounding, localization, action generation, closed-loop recovery, and temporal behavior.

Conclusion

This project helped me build a practical understanding of Vision-Language-Action policy learning for manipulation. Starting with a small 2D environment made the research process easier to control. I could change one idea at a time, inspect failures visually, and compare models without the complexity of a full robot simulator.

The main lesson is that robotic policy learning needs more than a model that performs well on a supervised validation set. A useful policy must work in closed loop, recover from imperfect states, and keep the language instruction connected to the action sequence. The exported results show this clearly: prediction errors improved, but task success remained a harder problem.

This makes the later temporal direction especially important. Manipulation is not only about choosing the next action from the current image. It is about understanding task progress over time. A policy needs to know whether it is approaching, grasping, moving, placing, or recovering from a mistake.

The value of this project is that it creates a clear research foundation. It starts from simple language-conditioned reaching, adds better visual grounding, tests different multimodal fusion methods, and then moves toward temporal manipulation policies. That progression gives me a structured way to study VLA models before scaling to larger datasets and more realistic robotic tasks.