How to collect training data for vision-language-action (VLA) models

June 10, 2026

Vision-language-action (VLA) models learn to map what a robot sees and is told to do into the actions that accomplish a task. Unlike a perception model that only needs labeled images, a VLA model needs demonstrations: synchronized streams of camera input, instructions, and the actions a competent operator took in response. Collecting that data well is the hard part, and it is mostly a hardware and operations problem, not a modeling one.

What good VLA training data looks like

A useful demonstration captures four things in lockstep:

The visual scene from the robot’s own cameras, at the resolution and frame rate the policy will run at.
The instruction or goal for the episode.
The operator’s actions, recorded as the same action space the robot will use (joint targets, end-effector poses, or gripper commands).
Accurate timestamps so vision and action stay aligned to within a few milliseconds.

Increasingly a fifth stream belongs on that list: proprioception and contact. Joint torques, gripper force, and tactile signals are still missing from most public datasets, and their absence is measurable. In one reported comparison on contact-rich tasks, a tactile-equipped policy succeeded 72 percent of the time against 30 percent for a vision-only policy on the same task. If your product involves insertion, wiping, folding, or anything where the interesting information is in the contact rather than the picture, plan to record force from the start. It cannot be recovered later.

Coverage matters more than raw volume. A model trained on ten thousand near-identical pick-and-place clips generalizes poorly. The same budget spent across varied objects, lighting, backgrounds, and recovery behaviors produces a policy that holds up in the real world.

Why teleoperation is the workhorse

Teleoperation, where a human operates the robot in real time while every signal is recorded, is the most reliable way to produce on-distribution demonstrations. Because the human drives the actual robot through its real cameras and actuators, the data already matches the hardware the policy will deploy on. That removes the simulation-to-real gap that plagues synthetic data.

The catch is throughput. The quality of your data is bounded by the quality of the teleoperation rig: low-latency control, comfortable operator ergonomics, reliable time-sync, and cameras placed where the policy will actually need to see.

Choosing a teleoperation modality

There is no single teleoperation setup. There is a family of them, and the choice is not a matter of convenience. Each modality shapes the action distribution you record, which means it shapes what the resulting policy can and cannot do. Pick it deliberately and early.

Leader-follower robot twins

The most direct arrangement is a kinematically isomorphic pair: a leader arm the operator physically backdrives, and a follower arm that mirrors it. The joint angles of the leader are the action labels for the follower, so there is no retargeting step and no mapping error to model. Bimanual versions of this idea, exemplified by the open-source ALOHA reference design, made low-cost dexterous bimanual data collection practical for academic labs and set the template much of the field now builds on.

The strengths are real. Correspondence is exact, operator training time is short because the mapping is physical rather than learned, and the recorded action space is already the deployment action space.

The weaknesses are structural. You need one leader rig per follower robot, so collection throughput scales linearly with capital equipment and floor space. Most low-cost implementations are unilateral, meaning the operator feels nothing of what the follower feels, which is exactly the wrong property for contact-rich tasks. Bilateral variants that close the force loop exist and are worth the extra engineering when contact matters, because without them operators regulate contact by watching rather than by feeling, and that produces visibly different and generally worse contact behavior in the data.

VR and AR headset teleoperation

Putting the operator in a headset and tracking their hands or controllers decouples the interface from any one robot. It is inexpensive, it works across embodiments, and consumer XR hardware now provides centimeter-scale global pose out of the box.

The cost is the retargeting problem. A human hand and a parallel-jaw gripper do not share an action space, so something has to map between them, and every mapping introduces artifacts that end up in the labels. Headset teleoperation also gives the operator no force feedback and a field of view that is not the robot’s, both of which change behavior in ways that are easy to overlook and hard to undo.

Humanoid whole-body teleoperation

Humanoids raise the difficulty considerably. Now the operator’s whole body is the input device, captured either by an instrumented suit, an optical volume, or in lighter-weight systems by a single camera and a pose estimator. The captured motion has to be retargeted onto a machine with different limb lengths, different mass distribution, different joint limits, and a balance problem the human solves unconsciously and the robot does not.

The practical consequence is that a large fraction of captured human motion is simply infeasible for the target robot and has to be filtered out before it can be used, which published whole-body retargeting pipelines handle by fitting a body model to the humanoid’s structure and then rejecting motions a control policy cannot realize. The operator, meanwhile, gets no vestibular feedback from the robot’s actual balance state, so they can command motions that feel fine and are not.

Hybrid AR with the robot’s own end effector

One of the more interesting recent directions removes the robot from data collection entirely. The operator carries a handheld version of the exact gripper the robot will use, instrumented with its own camera and IMU, and wears AR glasses for task framing and feedback. The Universal Manipulation Interface is the canonical published example: a handheld parallel-jaw gripper with a wide-angle camera as its only sensor, using visual-inertial SLAM for pose and fiducials for continuous gripper width.

The appeal is decisive. Because the end effector and camera viewpoint are shared with the robot, the observation and action spaces transfer. Because no robot is in the loop, collection scales with the number of operators and devices rather than the number of robots, and it can happen in real environments instead of a cell. That is a fundamentally different cost curve.

The costs are specific and worth stating plainly. You inherit visual SLAM’s dependence on scene texture, so low-texture environments degrade tracking. Latency has to be measured and compensated separately for observation and execution, since the handheld device and each target robot have different delays. And because the deployment robot’s kinematic limits are unknown at capture time, feasibility can only be enforced by filtering afterward, which means some fraction of collected data is discarded. Budget for that yield loss rather than being surprised by it.

Exoskeletons and isomorphic wearables

A wearable linkage that matches the robot’s kinematics sidesteps retargeting mechanically rather than computationally, and gives you drift-free joint angles straight from encoders. It is the most accurate option available for the degrees of freedom it covers. It is also the most constraining to wear, the least adjustable across different operator body sizes, and the most likely to cause fatigue over a long session. That trade is often worth making for high-precision dexterous work and rarely worth making for high-volume general collection.

How you track the operator

Underneath every modality above is a tracking question, and the honest summary is that no single sensor meets all the requirements at once. The useful mental model is to separate global pose, which is where the hand or body is in the room, from articulation, which is what the fingers and joints are doing. These have different failure modes and generally want different sensors.

Outside-in tracking puts the reference infrastructure in the environment. Optical motion capture with passive or active markers gives sub-millimeter accuracy with no drift, at installation costs that run from tens of thousands of dollars into the hundreds of thousands. Lighthouse-style base stations are far cheaper and achieve comparable static accuracy, but published measurements put dynamic error around ten millimeters, which matters because demonstrations are motion, not poses. Both approaches share the same two hard limits: they require line of sight, and they only work inside the instrumented volume. If your data collection has to happen in a customer’s warehouse or across many sites, that volume constraint is disqualifying.

Inside-out tracking moves the sensing onto the operator: cameras and IMUs running visual-inertial odometry or SLAM. There is no infrastructure to install and no volume boundary, which is why it is the backbone of in-the-wild collection. The trade is accuracy and drift. Expect centimeter-scale global accuracy at best, with error that accumulates over an episode and depends on scene texture.

For articulation the options diverge again. IMU-based gloves give good joint angles, roughly half a degree in pitch and roll, but drift in yaw without a magnetic reference. Electromagnetic finger tracking achieves millimeter-level accuracy and is immune to occlusion, which is exactly its appeal, since optical finger tracking is defeated by self-occlusion during grasping, precisely the moment you most need it. But EM systems are distorted by nearby metal, which is common in industrial settings, and EM fields overwhelm IMU magnetometers. That last point is a genuine incompatibility rather than an integration inconvenience: you can have drift-free yaw or you can have EM finger tracking, not both in the same glove.

The pattern that works in practice is hierarchical fusion, assigning disjoint degrees of freedom to the sensor best suited to each. Inside-out SLAM or a consumer headset handles global wrist pose, while gloves, electromagnetic tracking, or encoders handle the fingers. Nothing is asked to do a job it is bad at, and the fusion has no contested overlap to resolve.

Whatever combination you choose, characterize it before you collect at scale. Static accuracy from a datasheet is not the number that matters. Measure dynamic error at the speeds your task actually involves, measure drift over a full episode rather than a few seconds, and measure the time offset between every stream. Unmodeled latency between tracking and imagery does not announce itself in the data. It quietly appears later as a policy that is confidently wrong about when to close the gripper.

Human factors are a data quality problem

It is tempting to treat operator comfort as a duty-of-care matter that sits beside the technical work. For VLA data collection it is the technical work. Operator state is directly upstream of the action labels, so fatigue does not just slow collection down, it changes what the model learns. A tired operator moves differently, hesitates in different places, and recovers from errors differently, and the model faithfully absorbs all of it.

The magnitude is not marginal. A published ergonomic comparison of teleoperation interfaces found that a seated interface produced roughly 26 percent faster operator learning and 30 percent faster task execution than a standing whole-body motion capture interface, with NASA-TLX physical demand scores of 3.54 against 9.81 and 2.6 times less whole-body center-of-mass motion. Those are the same demonstrations, from the same people, differing only in how the interface asked them to hold their body.

This is compounded by the fact that operator-to-operator variation is already a recognized source of dataset variance, alongside embodiment and camera placement differences. If your operators are also drifting within a session, you are stacking an uncontrolled temporal variable on top of an uncontrolled between-subjects one.

Treat it as an experimental design problem and run real user studies before committing to a rig:

Instrument workload with a validated measure such as NASA-TLX rather than asking people if they feel fine.
Measure posture and range-of-motion comfort quantitatively. Kinematic metrics for posture deviation, joint usage, and center-of-mass divergence can be computed from the same tracking data you are already collecting, which makes this nearly free.
Characterize the degradation curve over a full shift, not a ten minute trial, and set session lengths from where the curve bends rather than from what fits the schedule.
Verify that the rig accommodates the range of body sizes you intend to hire across. A fixture that fits the person who designed it is a recruiting constraint disguised as a hardware choice.
Include recovery from failure in the study protocol. It is the highest-stress part of the task and it is the data you most need.

Log operator identity and session timing as metadata on every episode. You will want to be able to ask later whether a behavior in the policy came from the task or from the person, and you cannot ask that question retroactively if you did not record it.

Designing a rig that scales

A few principles separate a rig that produces clean data from one that produces noise:

Put the camera where the sensor will live. Train on the same viewpoint you will deploy on.
Keep control latency low, and measure observation latency and execution latency separately. Lag changes how operators behave and corrupts the action labels.
Hardware time-sync, not software guesses. Align camera and action streams at the source.
Build for long sessions. Operator fatigue shows up directly in data quality.
Capture failures and recoveries, not just clean successes. Policies need to learn how to get back on track.
Record more than you think you need, including force, proprioception, and full calibration state per episode. Storage is cheaper than recollection.

Most teams underestimate how much custom hardware this takes. Off-the-shelf arms and webcams get you a demo; collecting millions of high-quality, on-distribution demonstrations takes purpose-built rigs.

The open problems

Everything above is engineering that can be done today. It is worth being clear-eyed about what cannot, because the field’s remaining bottlenecks are not evenly distributed and it is easy to over-invest in the wrong one. These are the items that would genuinely change the economics of VLA development if they were solved.

A physics simulator good enough to close the loop on. This is the one most people name first, and correctly. If a simulator modeled contact, friction, compliance, and deformation faithfully enough, you could collect a small real dataset, learn a policy, and then improve it with reinforcement learning in simulation without the sim-to-real gap eating the gains. Rendering is no longer the obstacle. Contact dynamics are. Rigid-body approximations of soft fingers, cloth, cables, granular material, and friction under real surface conditions remain the failure point, and every survey of simulators for embodied AI lands on the same conclusion, that constructing simulations that are real enough for both training and evaluation is still unsolved. Note that this problem has a much harder and a much easier version. Faithful simulation of one known task in one known cell is tractable now. Faithful simulation of arbitrary contact in an unmodeled environment is not close.

Automatic real-to-sim. Even a perfect simulator is only useful if you can get your actual scene into it cheaply. Today building a matched simulation environment is skilled manual work, which limits how many environments anyone can afford to model and therefore caps the diversity of anything trained inside. Reconstructing geometry, materials, articulation, and physical parameters from a short scan of a real workcell, accurately enough to train against, would unlock the rest.

World models that respect physics. Generative world models can already produce visually convincing rollouts, and there is real progress in using them to expand a small number of real rollouts into many counterfactual variants, with reported reductions in required real interaction of over half on single-task setups. The unresolved issue is that visual plausibility is not physical validity. A rollout that looks right but ignores contact, force, friction, or stability is not supervision, it is a confident hallucination with a good frame rate. Grounding generated experience in physical consequence, and knowing when to trust it, is an active and unfinished problem.

Evaluation. This is underrated and may be the tightest constraint of all. Robotics has no held-out test set. Evaluating a policy means running it on hardware, in a specific scene, enough times to be statistically meaningful, and the field currently evaluates on a small number of robot types in a small number of predefined environments. Simulated benchmarks help but inherit exactly the fidelity gap above. Until evaluation is cheap, reproducible, and comparable across labs, progress is hard to measure and therefore hard to direct, and published improvements are hard to distinguish from evaluation noise.

A genuinely cross-embodiment action space. Action heterogeneity across morphologies still prevents zero-shot transfer. Universal action codebooks are a promising direction and have not yet delivered true zero-shot generalization to robots with different degrees of freedom or structure. Solve this and every dataset ever collected becomes usable for every robot, which is a step change in the value of all the collection work described in this article.

Turning human video into robot supervision. There is vastly more footage of humans doing physical tasks than there will ever be of robots doing them, and almost none of it carries embodiment-specific action labels, task semantics, or reward structure. Systems that automatically extract object state, contact events, task phases, and rewards from unstructured human video and wearable data would change the supply curve for training data more than any improvement in teleoperation throughput. It is also the hardest of these problems, because it requires inferring physical quantities that were never measured.

Deployment loops that improve themselves. Fleets currently treat deployment as evaluation. A robot that fails a task in the field, and turns that failure plus a human correction into training signal that improves the fleet, is the mechanism that would let data collection compound instead of being a fixed capital expense repeated per product.

Force and touch as first-class modalities. Related to the first item on this list but distinct from it. Most VLA work remains vision-centric, and the tactile results cited earlier suggest that is leaving substantial performance unclaimed on contact-rich tasks. This one is arguably the most tractable item here, and it is largely a hardware, calibration, and data-collection problem rather than a modeling one, which is to say it is the one a team can start on this quarter.

If you are planning a data collection program, the practical read is that the simulator and world-model items are worth tracking rather than betting on, while sensing, force, calibration, evaluation discipline, and human factors are all available to you now and are where the near-term wins are.

Where AutoRoboto fits

We design the camera-driven hardware that makes this possible: teleoperation rigs across the modalities above, operator tracking and sensor selection, camera and full-system calibration, force and tactile integration, human factors studies to validate a rig before you scale it, and the data pipelines that turn raw sessions into clean, validated training sets. All of it built and commissioned at your facility, on your hardware.

If you are standing up a VLA data-collection program, or deciding which teleoperation approach fits the product you are actually shipping, get in touch.