Robotics is having its scaling moment. The success of large language models has convinced much of the field that the same recipe (collect enormous datasets, train one large end-to-end network, watch capability emerge) will carry over to physical machines. Vision-language-action models (VLAs) such as RT-2, OpenVLA, and π0 are the flagship products of this belief[1]. In parallel, a second camp is betting on world models: networks like Meta's V-JEPA 2 that learn to predict how a scene will evolve, then plan against that prediction[6]. Both directions have produced genuinely impressive results. We believe both, in their current form, are also incomplete, and the argument for why points toward a different architecture for the coming decade of robot learning.
This technical blog lays out that architecture, the one we are building at Geodesic. The short version: internet-scale video and demonstration data are extraordinarily valuable, but not as the direct substrate of a control policy. Their proper role is to train priors: reusable knowledge about how humans move, grasp, and interact, which are then injected into increasingly capable physics simulators, where reinforcement learning (RL) forges them into robust controllers. And crucially, those controllers should not be frozen at deployment. When a robot errs in the world, a high-level model should diagnose the failure, the simulator should update its physical parameters to match reality, and the policy should be rapidly fine-tuned: a control loop that keeps the policy fluid for its entire operational life.
Section 01The scaling hypothesis meets the physical world
The scaling argument goes like this: image models scaled when we had enough images; language models scaled when we had enough text; therefore robot policies will scale once we have enough robot data. The analogy is seductive, but it hides an asymmetry of many orders of magnitude. Frontier language models train on trillions of tokens harvested passively from the internet. Robot action data, by contrast, must be manufactured: teleoperated demonstration by demonstration, robot hour by robot hour. Even the largest community efforts, such as Open X-Embodiment, contain on the order of a million trajectories across a couple dozen robot types[1], tiny compared to the corpora that made LLMs work, and a widely acknowledged bottleneck: surveys of scaling in robotics identify the scarcity of internet-scale robot data as the central barrier to generalizable models[2].
There is careful empirical work on whether data scaling laws exist in manipulation at all. Lin et al. collected over 40,000 demonstrations and ran more than 15,000 real-world rollouts to measure how imitation-learning performance grows with data[3]. They did find power-law improvements, but the operative variable was diversity of environments and objects, not raw quantity, and the result held for single-task policies within an object category. That is an important and optimistic finding, but it is a much narrower claim than "end-to-end scaling yields general physical intelligence." The gap between those two claims is where we believe the field's current bet is misplaced.
The evidence on brittleness
If end-to-end VLAs were quietly generalizing, robustness benchmarks would show it. They show the opposite. LIBERO-Plus, a fine-grained robustness analysis of leading VLAs, concludes that apparent zero-shot success in familiar settings largely reflects interpolation within the training distribution rather than true generalization[4]. VLATest, a systematic testing framework applied to seven representative VLA models, found that current models lack the robustness necessary for practical deployment once scenes, perception, or language deviate from training conditions[5]. Work on robustness-aware post-training reports the same pattern from another angle: VLAs frequently fail under exactly the disturbances that are unavoidable in deployment: observation noise, sensor error, actuation perturbation[12].
None of this means VLAs are useless, far from it. It means that an end-to-end mapping from pixels and words to motor torques, learned by imitation, inherits every limitation of its training distribution and has no built-in mechanism to recover when the physical world drifts outside it. A policy that is "arbitrary out of distribution," to put it plainly, is not a foundation you can deploy a humanoid on.
World models: right instinct, wrong prediction target
The world-model camp starts from a sounder instinct: an agent should carry an internal model of how the world evolves. Meta's V-JEPA 2 is the strongest recent expression of this idea. It is pretrained on more than one million hours of internet video, and remarkably, its action-conditioned variant (V-JEPA 2-AC) needed less than 62 hours of unlabeled robot video to enable zero-shot planning for pick-and-place on Franka arms in labs it had never seen[6]. Unlike pixel-generating video models, JEPA predicts in an abstract latent space rather than reconstructing every frame, which is precisely why it generalizes better and plans dramatically faster than diffusion-based world models like Cosmos in head-to-head comparisons[6].
We take this result seriously: it is the best evidence yet that passive video contains transferable physical knowledge. But look closely at what is being predicted: representations of future video frames. The latent space is still, fundamentally, an embedding of appearance from a camera's point of view. And that is our core disagreement with the image-prediction paradigm, whether in pixels or in latents:
We do not live in an image. We live in a spatial world. Physical interaction is governed by 3D geometry, contact, force, friction, mass distribution, and compliance: quantities that are only indirectly and lossily encoded in how a scene looks. The empirical fingerprints of this mismatch are visible in V-JEPA 2-AC's own reported limitations: manipulation success rates of roughly 60–80% (well below what deployment demands), and a sensitivity to camera pose: performance degrades roughly linearly with azimuth error in camera placement[6]. A model whose "understanding" of the world shifts when you move the camera has learned something about viewpoints, not about the world. Studies comparing latent video world models against ground-truth physics likewise note that such models are purely vision-based and not physically constrained[7]. That is a hallucination risk, not just an accuracy gap: a model trained to predict pixels or latents can imagine any future that looks plausible, whether or not it is physically possible.
Predicting the future in appearance space is a detour. The future of a physical scene is most naturally represented in state space (poses, velocities, contacts, material parameters), which is exactly the representation a physics simulator natively operates in.
The reframe: video is a prior, not a policy
Here is the constructive claim. The million hours of humans doing things on video are not a control dataset. They are a prior: a dense record of what plausible, purposeful physical behavior looks like: how bodies balance, how hands approach objects, how tools are held, in what order subtasks unfold. The right question is not "how do we regress actions from this video," but "how do we distill this knowledge into a form a physics-grounded learner can exploit?"
The field has, in fact, already begun answering that question, under the banner of retargeting. A rapidly maturing line of work converts ordinary human video into structured reference material for robots:
Whole-body humanoid motion. H2O retargets large-scale human motion capture (SMPL-format motion) onto a humanoid's body, filters out physically infeasible sequences with a privileged "sim-to-data" policy, and trains a robust imitation controller in simulation that transfers zero-shot to a real full-sized humanoid for real-time teleoperation[8]. VideoMimic goes further: from a casually captured phone video it reconstructs the 4D human-and-scene geometry, retargets the motion to a humanoid inside the reconstructed scene, and trains RL policies to track it, turning everyday video directly into simulator-ready training material[9].
Manipulation and interaction. HDMI extracts human and object trajectories from unconstrained monocular RGB video, retargets them, trains an RL policy to co-track robot and object states, and deploys zero-shot on real humanoids[10]. Video2Sim2Real reconstructs a digital-twin simulation from a single human manipulation video, refines the retargeted trajectory in simulation, and bridges the remaining gap with residual RL[11]. ASAP begins its pipeline the same way: recording human videos of agile motions, reconstructing 3D motion, and retargeting it to create imitation goals for motion-tracking policies[13].
Notice the common shape of all of these systems: video in, prior out; simulator plus RL in the middle; physically grounded policy at the end. The video contributes what it is genuinely good at (the space of natural, purposeful motion), while the simulator contributes what video cannot: contact forces, torque limits, mass, friction, and millions of trials of consequence-bearing practice. VLAs and video world models have a place in this picture too, as semantic and behavioral priors, proposing goals, decomposing tasks, scoring progress, rather than as the final motor controller.
Section 04The simulation substrate is maturing fast
This architecture is only as strong as its simulators, and this is where the trend lines are most encouraging. Two developments stand out.
Throughput. NVIDIA's Isaac Lab, successor to Isaac Gym, provides GPU-native parallel physics, photorealistic tiled rendering, actuator models, multi-frequency sensor simulation, and built-in domain randomization over both physical parameters (friction, mass, joint properties) and visual ones (textures, lighting)[14]. State-based environments reach hundreds of thousands of simulation steps per second on a single machine, and the framework is moving toward the differentiable, GPU-accelerated Newton physics engine[14]. Genesis pushes throughput further still, reporting up to 43 million FPS when simulating a Franka arm on a single RTX 4090 (roughly 430,000× faster than real time), with locomotion policies trainable in tens of seconds[15].
Fidelity and coverage. The historical objection to simulation, that "it only does rigid bodies," is expiring. Genesis unifies rigid-body, MPM, SPH, FEM, PBD, and fluid solvers in one engine, allowing coupled simulation of liquids, gases, cloth, granular media, and deformable objects alongside articulated robots[15]. Its 2026 "Genesis World" release focuses on high-fidelity unified rigid-and-deformable physics with major gains in speed and numerical stability, explicitly positioned as scalable, closed-loop evaluation and training infrastructure for robot foundation models[16]. Surveys of embodied-AI simulation similarly place Genesis and the Isaac family at the high-fidelity end of deformable-object support[17]. And unlike a learned predictor, a physics engine cannot hallucinate: every rollout obeys the same conservation laws and contact rules, so a policy trained in it only ever meets an imperfectly calibrated world, never an impossible one.
Every year, the set of physical phenomena that can be simulated faster than real time, at scale, in parallel, with randomization, grows. Every such gain compounds the value of the priors-into-simulation architecture, because domain randomization over thousands of parallel worlds is exactly the mechanism that converts a narrow prior into a robust policy.
Section 05The missing piece: policies that stay fluid after deployment
Even a policy trained across millions of randomized worlds meets a reality it has never seen. The standard answer, make the randomization range wider, trades peak performance for conservatism and still fails under drift the designer didn't anticipate: actuator heating, wear, payload changes, a floor that got mopped. Surveys of sim-to-real transfer are blunt about this: policies remain robust only within the distribution of conditions seen in training, and deployment inevitably introduces residual mismatch[18].
The answer, we argue, is that deployment should not be the end of learning. The deployed policy should sit inside a closed adaptation loop with three coupled mechanisms operating at different timescales:
1 · Fast, implicit adaptation (milliseconds–seconds). This already works. Rapid Motor Adaptation (RMA) trains a base policy conditioned on a latent encoding of the environment's physical properties; at deployment an adaptation module infers that latent from the last fraction of a second of proprioceptive history, effectively online system identification in latent space, letting quadrupeds adapt to slippery, rocky, and deformable terrain in real time with no fine-tuning[19]. Extensions like A-RMA further refine the base policy with model-free RL[20].
2 · Semantic failure diagnosis (seconds–minutes). When something goes wrong at the task level (the grasp keeps slipping, the foot placement is subtly off), a high-level model should say so, and say why. This is where VLAs and vision-language models earn their keep in our architecture: not as motor controllers, but as supervisors. Recent systems make this concrete: ReCoVLA uses a VLM failure detector to classify what went wrong in a manipulation rollout and compiles that diagnosis into rewards for a corrective residual policy[21]; deployment monitors like RAPT detect out-of-distribution execution online for humanoids and run automated root-cause analysis, precisely because sim-trained policies otherwise fail silently in OOD states[22].
3 · Simulator re-calibration and rapid fine-tuning (minutes–hours). The diagnosis then flows into the simulator. Real observations are used to update the simulator's physical parameters: friction coefficients, contact and compliance properties, actuator characteristics, so that the training world tracks the deployment world. This "real-to-sim-to-real" loop also has a growing evidence base: auto-tuned sim-to-real adjusts simulator system parameters directly from raw RGB observations of the real robot[23]; Real-Sim-Real loop frameworks iterate between collecting real data, tuning a differentiable simulator, retraining, and redeploying[24]; Swim2Real shows that a VLM can itself propose simulator parameter corrections, calibrating sixteen coupled physical parameters by comparing simulated and real videos[25]; and ASAP learns a delta-action model from real rollouts specifically to align simulator physics with real-world physics before fine-tuning agile humanoid skills[13]. Phys2Real fuses VLM-derived physical priors with online interactive adaptation for uncertainty-aware manipulation transfer[26].
Individually, each of these mechanisms exists and is published. The architectural claim of this technical blog is that they belong together, permanently, around every deployed policy: a robot whose controller is never frozen, only ever between fine-tunes. This is the system Geodesic is building.
Anticipating the objections
"Simulation will never match reality." It doesn't have to match reality perfectly, that is precisely what the adaptation loop is for. The sim-to-real gap is treated not as an error to eliminate in advance but as a signal to consume continuously: real rollouts recalibrate the simulator, and the simulator amortizes that correction over millions of cheap trials[13][24]. Meanwhile the fidelity frontier keeps moving: deformables, fluids, granular media, and differentiable contact are now in mainstream engines[15][17].
"VLAs will fix robustness with more data." Perhaps partially: the data-scaling results are real within their scope[3]. But the scope matters: those power laws were measured for single tasks with diversity as the driver, and the robot-data supply grows linearly with human effort, not exponentially like web text. Betting the entire stack on out-collecting the physical world's variability is a bet against combinatorics.
"Latent world models already avoid the pixel problem." Predicting in latent space is a real improvement over generating pixels, and V-JEPA 2's data efficiency is a landmark[6]. But a latent of camera observations still inherits the camera's frame: the reported viewpoint sensitivity is the symptom[6]. Our position is not that these models are wrong, but that their proper home is upstream (as priors, monitors, and progress estimators) and alongside (as failure detectors[21]), while the substrate of prediction for control should be explicit spatial state, which simulators provide for free.
"Isn't this just more engineering, less learning?" No: it is learning placed where each kind of data is strongest. Passive video teaches what to do; simulation teaches how to do it under physics; deployment teaches where the model of physics was wrong. Each stage is learned; only the interfaces are designed.
Section 07Conclusion: fluid machines
The end-to-end dream asks a single network to be perception, physics, and controller at once, and to have anticipated every deployment condition at training time. The evidence (brittleness benchmarks[4][5], the data-supply asymmetry[2], viewpoint-fragile world models[6]) suggests that is the wrong division of labor for the physical world.
The alternative sketched here has three commitments. First, treat internet-scale video and demonstration data as priors: the retargeting literature already shows how to turn ordinary video into simulator-ready knowledge[8][9][10]. Second, make simulation the forge: GPU-parallel, randomized, increasingly deformable-and-fluid-capable engines are where priors become policies[14][15]. Third, never freeze the policy: fast latent adaptation[19], semantic failure diagnosis[21][22], and continual simulator re-calibration with rapid fine-tuning[23][24][25] keep it aligned with a drifting world.
Biological motor control works this way: evolution and observation supply the priors; practice under real physics supplies the skill; and the skill is re-tuned for as long as the organism lives. Robots that matter will be built the same way: not frozen artifacts of a training run, but fluid machines in permanent conversation with the physics around them.
This is the architecture we are building at Geodesic: priors distilled from passive data, policies forged in simulation, and a permanent adaptation loop that keeps every deployed system learning.