Generating Visual-Language Robot Episodes with Robosuite

Project status: ongoing. This article will be updated as the simulator, dataset format, and policy-learning workflow mature.

Why I Started This Project

This is an ongoing project log for my robosuite-based manipulation simulator. The goal is to build a cleaner and more reliable way to generate visual-language-action episodes for robot policy learning.

The project uses MuJoCo for physics and robosuite for robot, controller, task, and observation handling. The first benchmark is intentionally narrow: a Franka-class robot arm stacks one block on another. Starting with a simple task makes it easier to focus on the parts that matter for policy-learning data: stable resets, clear language instructions, visual observations, action labels, success and failure labels, and reproducible episode recording.

From Demo Simulation to Dataset-Grade Simulation

A visual robot demo is not the same as a useful dataset. A demo only needs to look correct once. A dataset needs to be repeatable, measurable, and honest about what happened.

For this project, I care about recording not only images and robot actions, but also the meaning of each episode. Did the robot grasp the object? Did it place the object correctly? Did it fail, and if so, how? These labels matter because future learning models should not learn from unclear or misleading demonstrations.

Why Visual-Language Episodes Matter

The long-term goal of this project is to support robot policy learning with vision-language models and vision-language-action systems. In this setting, the robot does not learn only from numbers or isolated images. Each episode connects what the robot sees, what it is asked to do, and what action it takes over time.

For example, an episode may include the instruction “move the red block onto the purple block,” a sequence of RGB images from the simulated camera, robot state values, object positions, gripper commands, and success labels. This combination makes the data more useful for models that need to connect language instructions with physical robot behavior.

This is also why I am treating the simulator as a data pipeline, not just a visual demo. A robot-learning dataset should make the relationship between instruction, perception, action, and outcome easy to inspect.

Why Robosuite

Robosuite provides a more structured starting point for manipulation research than maintaining every piece of simulation code manually. It gives access to robot models, controllers, cameras, observations, and task abstractions while still using MuJoCo as the physics engine.

In this project, robosuite helps keep the focus on the dataset and task design instead of low-level simulator glue. The project code can then focus on the parts that are specific to my research goals: block stacking, scripted policies, success checks, episode metadata, and dataset writing.

Current Task: Franka Block Stacking

The current task uses a fixed-base Franka-class arm in a tabletop scene. The robot must pick one block and place it on another block. The task is simple, but it contains the core structure of many manipulation problems: reaching, grasping, lifting, transporting, placing, and checking the final result.

The current implementation supports scripted demonstrations, video generation, live viewing, and dataset export. There are two dataset paths: a simpler JSONL format for quick inspection, and a richer temporal episode format for later policy learning.

What One Episode Contains

The richer dataset format stores full episodes rather than isolated images. Each episode combines language, vision, state, actions, and outcome labels.

Example Episode Metadata

One useful part of the project is that each recorded episode stores a compact metadata file. For example, one successful episode records the instruction:

move the red block onto the purple block

The metadata also records the task type, source and target colors, available distractor blocks, camera name, success labels, and number of steps. In this example, the episode was successful: the robot grasped the red block, placed it on the purple block, did not drop the object, and completed the task in 335 recorded steps.

This metadata is important because the image frames alone do not explain what happened. The metadata connects the visual rollout to the task instruction, the action sequence, and the final outcome. That makes the dataset easier to inspect, debug, and use for later learning experiments.

Dataset Design

The current output structure is:

3D_PP_VLA_dataset/
├── manifest.json
├── splits.json
└── episodes/
    └── episode_000000/
        ├── meta.json
        ├── steps.jsonl
        └── frames/

This design keeps the temporal structure of the task. A future policy can learn not only from a single image, but from how the scene changes as the robot approaches, grasps, moves, places, and returns.

Relevance to Robot Policy Learning

Modern robot-learning work increasingly expects more than model training. The practical loop includes data collection, simulation, training, evaluation, failure analysis, and deployment-oriented thinking. This project is a small version of that loop.

The robosuite environment gives a controlled simulator foundation. The generated episodes provide language-conditioned demonstrations. The metadata makes the data easier to filter and audit. The phase labels help diagnose where behavior breaks down. The success labels make task completion measurable.

This kind of structure is useful for imitation learning, reinforcement learning, and VLA-style experiments because it keeps the policy target connected to the real task. The model should not only see an image or read an instruction. It should learn how observations, language, actions, and outcomes fit together across time.

Current Progress

At the current stage, the project can generate temporal 3D pick-place episodes with colored blocks, distractor objects, instructions, action labels, phase labels, and success or failure outcomes. The first dataset path has already produced a 200-episode dataset for experimentation.

The current article images are selected from one successful high-resolution preview episode generated inside the project devcontainer. They show the initial tabletop scene, the robot approaching the source block, the placement phase, and the final stacked state.

What I Am Learning

The main lesson so far is that dataset generation is not just about rendering images. The difficult part is building a trustworthy connection between the task, the robot behavior, the labels, and the exported files.

A dataset for robot learning needs clear answers to practical questions: Was the grasp valid? Was the placement stable? Did the robot fail because of control, contact, object placement, or timeout? These details are easy to ignore in a demo, but they matter when the data is used to train or evaluate a model.

Next Steps

The next steps are to improve environment version pinning, strengthen success and failure checks, and connect the dataset more directly to temporal policy-learning experiments.

I also plan to keep the article updated as the project matures, especially when the simulator becomes reliable enough for larger-scale dataset generation and model training.