Perceptron Introduces Isaac 0.5: An Open-Source Embodied Foundation Model

Perceptron, a foundation model company building physical intelligence, led by Armen Aghajanyan and Akshat Shrivastava, has released Isaac 0.5, its new open-source embodied foundation model. To their knowledge, it is the first open model at the frontier of multimodal video understanding, embodied reasoning, and robot control.

Isaac also establishes a scaling law for balancing general video, egocentric, UMI, and teleoperation data. It provides a framework for selecting the right mix of video and robot demonstrations as training scales, given a fixed budget and target action loss.

Scaling general video from 1,000 to 1 million hours reduced the teleoperation data needed to reach a well-calibrated held-out action loss from roughly 5,900 hours to just 28.

Isaac is a 36-billion-parameter sparse model that processes images, video, language instructions, robot state, and previous actions. It can understand videos, locate and track objects, assess task progress, and generate robot actions. Teams can fine-tune it as a policy or use its visual outputs in planners, controllers, and data engines.

The model was trained on data from 35+ robot systems, 100K hours of robot experience, 1M hours of general video, and 3T multimodal tokens. Video understanding, spatial grounding, task progress, and robot action were co-trained from the start. They are releasing Isaac as a complete system: checkpoints, training code, and inference code via LeRobot. The technical report contains the full training and evaluation details.

A Scaling Law for Video and Robot Action Data.

Recent research shows that both data volume and diversity matter for reliable robot control. But the optimal balance between general video, egocentric data, and robot experience has remained largely unexplored.

Isaac’s experiments establish a scaling law showing that cheap video data can substitute for expensive teleoperation. Holding an 80:30:30 ratio of general video, egocentric, and UMI data, the team scaled pretraining and measured action-prediction loss on held-out robot trajectories.

More video consistently reduces the teleoperation required. At lower teleoperation volumes, increasing video tenfold reduces action loss by about 0.006; beyond roughly 100 hours of teleoperation, the reduction reaches about 0.21.

The result is a 210× reduction in teleoperation requirements. To reach an action loss of 2.50, 1,000 hours of video requires ~5,900 hours of teleoperation, while 1 million hours of video requires just 28 hours.

Teleoperation is shown on the horizontal axis and general video on the vertical axis, with each contour representing training mixtures that achieve the same held-out action loss.

General video provides broad coverage of objects, activities, and environments. Egocentric video adds task-relevant views of hands, motion, and contact. UMI connects visual knowledge to manipulation, while teleoperation grounds these capabilities in a complete embodiment and action space.

Training all four sources together shapes the representation Isaac uses for action. The resulting scaling law provides a practical framework for planning data collection under a fixed budget.

Semantic World Modeling for Robotics.

Robot models can predict different kinds of futures, and the prediction target affects how well those representations transfer to control. Diffusion and pixel-reconstruction models predict future frames; joint-embedding models predict visual features; action models predict actions.

Isaac 0.5 adds semantic world modeling as a co-training objective, learning from video by predicting future percepts, task-relevant states, or changes such as object states, spatial relationships, affordances, task phases, and visible contact changes.

Percepts sit between pixels and actions. Rather than supervising them directly, Isaac learns from semantic descriptions automatically derived from future observations, including general video without human action labels.

Given observations up to time t, Isaac predicts a percept at a future time t + Δ. This trains the same shared backbone used for action generation.

The model co-trains video perception, embodied visual reasoning, future-percept prediction, and robot control, with every objective updating the shared backbone. The discrete FAST and continuous Flow decoders remain separate, but both operate on the same learned representation.

One Model, Several Useful Interfaces.

Isaac 0.5 unifies images, video, time, robot state, and past actions into a single sequence, jointly modeling across modalities.

Building a unified multimodal interface at this scale requires solving three challenges:

1. A common format for concurrent multimodal streams.

2. A high-throughput data stack capable of handling petabyte-scale datasets.

3. New architectures that support dynamic compute while jointly modeling perception and action.

Modeling images, video, and control trajectories together requires a consistent way to serialize multimodal inputs. Isaac introduces mHarmony, a typed compiler based on OpenAI’s Harmony that extends the format to multimodal event streams.

mHarmony keeps data consistent across data preparation, pre-training, inference, and RL pipelines. For training, these structures are converted into TensorStream, a packed multimodal tensor format designed for efficient processing.

Throughput-Optimized Data Stack.

Scaling to 1M+ hours of video exceeds the practical limits of NFS across datacenters. Running a full epoch locally is no longer feasible, requiring cloud streaming,  but network bandwidth, storage constraints, and CPU-heavy processing can limit MFU.

The training data stack uses a topology-aware predictive planner to efficiently stream batches during training while looking ahead and maintaining granular control over dataset mixtures.

Dynamic Multimodal Architectures.

Real-time control has traditionally relied on smaller dense models, limiting model scaling as a path to better performance. Mixture-of-Experts (MoE) enables larger total parameter counts while keeping inference costs tied to the number of active parameters.

Isaac 0.5 extends MoE with Null Experts, a paradigm where each token can select a variable number of experts. This allows compute to scale dynamically with task complexity, using more compute for harder tasks and less for simpler ones.

Compute maps reveal strong saliency patterns across the architecture, showing that the model dynamically allocates computation based on the task.

Joint Discrete and Continuous Actions.

Isaac 0.5 supports discrete and continuous control from the same shared backbone.

The discrete path uses FAST to compress action trajectories into tokens, allowing robot actions to be predicted through the same autoregressive machinery used for language and grounding. The continuous path uses a dedicated Flow expert to generate action chunks by transforming noise into trajectories conditioned on the scene representation.

Together, they provide two complementary interfaces: FAST offers a unified, token-based representation that integrates naturally with the VLM, while Flow preserves the precision and multimodality of continuous control.

Both rely on the same perception and reasoning backbone, allowing knowledge learned from images, video, and embodied reasoning to directly influence action generation.

Evaluation Across Perception and Control.

The same Isaac checkpoints are evaluated across video understanding, spatial grounding, action prediction, and closed-loop robot control, connecting perception and prediction to real-world behavior.

On grounding and counting, Isaac 0.5 scores 62.6 on ScreenSpot-Pro, 32.8 on LVIS Count, and 19.1 on CARPK, compared with 54.8, 28.7, and 6.0 for the strongest Qwen3-VL run under the same evaluation harness.

Across all five evaluated task families, physical and temporal reasoning, spatial and embodied reasoning, structured visual understanding, general visual intelligence, and grounding and counting, no open comparator matches Isaac. It also achieves these results at 8.5× lower inference cost: 26.9 TFLOPs for a three-image request versus 228.4 TFLOPs for the strongest comparator.

Shared weights also show up in how quickly Isaac adapts to new tasks. On a physical chess-manipulation benchmark, each policy is fine-tuned for one epoch on a single expert episode and tested on a held-out episode with a perturbed piece pose, different chess notation, or a different branch of the same opening.

Isaac 0.5 achieves the largest loss reduction at every level: 10.5× on the fixed move, 9.5× under notation conditioning, and 7.0× on the defensive line. π0.5 follows at 3.1×, 2.6×, and 2.3×, respectively.

Individual margins are close to seed-to-seed standard error, so the key result is the consistent ordering across all three levels. These are the same weights behind the perception results above.

Building with Isaac in the Open.

Open-source models can accelerate progress in robotics by giving researchers and developers a foundation to build on.

Isaac 0.5 is available as a starting point for embodied systems. It can be fine-tuned on custom demonstrations, integrated into larger systems through its visual outputs, and run locally through LeRobot.

𝐑𝐞𝐦𝐨𝐭𝐞 𝐇𝐢𝐫𝐞 𝐖𝐢𝐭𝐡 𝐔𝐬: 𝐌𝐞𝐧𝐥𝐨 𝐓𝐚𝐥𝐞𝐧𝐭 helps technology companies hire exceptional remote talent from India. Build your team with carefully curated engineers, product leaders, designers, GTM professionals, and more. Learn More At: https://www.menlotimes.com/menlo-talent