You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _posts/2025-11-19-walrus.md
+48-41Lines changed: 48 additions & 41 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
layout: post
3
3
title: "Walrus: A Cross-domain Foundation Model for Continuum Dynamics"
4
4
authors: Michael McCabe, Payel Mukhopadhyay, Tanya Marwah, Bruno Regaldo-Saint Blancard, Francois Rozet, Cristiana Diaconu, Lucas Meyer, Kaze W. K. Wong, Hadi Sotoudeh, Alberto Bietti, Irina Espejo, Rio Fear, Siavash Golkar, Tom Hehir, Keiya Hirashima, Geraud Krawezik, Francois Lanusse, Rudy Morel, Ruben Ohana, Liam Parker, Mariel Pettee, Jeff Shen, Kyunghyun Cho, Miles Cranmer, Shirley Ho
5
-
shorttitle: "WalrusFoundation Model"
5
+
shorttitle: "Walrus: A Cross-domain Foundation Model for Continuum Dynamics"
6
6
date: 2025-11-19 11:00
7
7
smallimage: walrus-splash.jpg
8
8
image: walrus-splash.jpg
@@ -19,10 +19,10 @@ Over the last few years, researchers have sought to side-step this data dependen
19
19
20
20
---
21
21
22
-
####Why is it so Hard to Build a Foundation Model for Physical Simulation?
23
-
22
+
## Why is it so Hard to Build a Foundation Model for Physical Simulation?
23
+
<br/>
24
24
Physical simulation is an enormously broad field. It can be difficult to even define what we mean by “physical simulation” since there are many varieties and scales that are regularly simulated. Here, we’re mostly speaking of “continuum-level” simulation where we’re simulating macroscopic objects as though they were coherent objects rather than astronomically large collections of molecules crashing into each other. But even at this particular level, there is still an enormous amount of diversity that must be accounted for:
Walrus is a transformer-based model designed specifically to learn across diverse physical systems. It contains **1.3 billion parameters** and is trained on a dataset of unprecedented scale and variety: **19 scenarios**, encompassing **63 physical fields**, drawn from areas including **acoustics, classical fluids, non-Newtonian flows, plasma physics, active matter**, and several **high-resolution astrophysical regimes**. Walrus is one of the largest, most broadly pretrained models yet for physical emulation.
44
45
45
46
---
46
47
47
-
#### The Anatomy of a Walrus
48
-
<palign="center">
49
-
<imgsrc="/images/blog/walrus/walrus_closeup.jpg"alt="Comparisons on downstream 2D challenges"width="95%"style="mix-blend-mode: darken;">
50
-
</p>
48
+
### The Anatomy of a Walrus
49
+
\
51
50
Walrus learns by watching large amounts of simulation data— movies of physical systems evolving over time. **Walrus takes a short trajectory of system snapshots and predicts the next state in the system. Rather than being explicitly provided information about the equations or system coefficients, Walrus must infer this information in-context from the provided history.** This allows Walrus to be used on experimental data or settings where there may not be a clean equation that models the system.
52
51
53
52
To make this possible over so many types of data, we had to build in a few ideas that help the model learn efficiently and stay accurate over long sequences:
Physical systems are sensitive: if you make a tiny mistake early on, that mistake can grow and completely change what happens later. Machine learning models can amplify this error due to architectural choices. For example, the “patching” or “tokenization” procedure used for compression in higher dimensional transformer models can break translation equivariance, the property of physical dynamics that says that outside of boundary effects physics should not depend on the location inside a domain. Walrus avoids this by randomizing the compression process. Before downsampling, Walrus randomly **“jitters”** the data, so that it reads the data slightly differently each step. **These tiny shifts prevent the model from locking onto grid patterns or numerical artifacts. The result is that Walrus stays stable for far longer than earlier models. **
Many physical systems are sensitive: a tiny source of error early can be magnified by the dynamics and completely change what happens later.
68
+
Machine learning models can amplify this error due to architectural choices.
69
+
For example, the “patching” or “tokenization” procedure used for compression in higher dimensional transformer models can break translation equivariance,
70
+
the property of physical dynamics that says that outside of boundary effects physics should not depend on the location inside a domain. Walrus avoids this by randomizing the compression process.
71
+
Before downsampling, Walrus randomly **“jitters”** the data, so that it reads the data slightly differently each step. **These tiny shifts prevent the model from locking onto grid patterns or numerical artifacts. The result is that Walrus stays stable for far longer than earlier models.**
Not all systems require the same amount of compute to emulate. Walrus is built with this in mind and **uses recently developed [compute-adaptive patching](https://arxiv.org/pdf/2507.09264) techniques to apply different levels of compression to different inputs**. Walrus can, for instance, apply less compression to already coarse-grained data while applying more to higher resolution data to scale each problem to the available compute to maximize accuracy.
73
+
This isn't just a heuristic. We can root this method in solid analysis of the operations used in these
74
+
models, but that's beyond the scope of this blog post, so read the paper if you want to know more.
75
+
76
+
#### Adaptive Compute Patching
71
77
72
-
**Dimensional Augmentation**
78
+
Not all systems require the same amount of compute to emulate.
79
+
Walrus is built with this in mind and **uses recently developed [compute-adaptive patching](https://arxiv.org/pdf/2507.09264) techniques to apply different levels of compression to different inputs**.
80
+
Walrus can, for instance, apply less compression to already coarse-grained data while applying more to higher resolution data to scale each problem to the available compute to maximize accuracy.
81
+
82
+
This helps us avoid some of the limitations of fixed resolution models, especially for 3D data where the trade-off between accuracy
83
+
and accuracy is especially impactful.
84
+
85
+
#### Dimensional Augmentation
73
86
<img
74
87
src="/images/blog/walrus/DimensionPadding.png"
75
88
alt="Treating 2D data as 3D for joint augmentation."
Another key idea is **treating 2D and 3D data in a unified way through shared augmentation**. The presence or absence of certain fields can allow models to easily learn to predict entirely different dynamics for 2D and 3D systems, but this defeats the purpose of joint pretraining. In training Walrus, we avoid this by an aggressive augmentation strategy in which all 2D data is embedded in a 3D space, sort of like placing a sheet of paper inside a thin box, and then randomly transformed with tensor law aware transformations in time and space so that the 2D data corresponds to a random 2D plane in the 3D space.
79
92
These design choices, that are described in further detail in the paper, let Walrus do something that hasn’t been possible before: **learn from extremely different kinds of physical systems— waves, fluids, plasmas, turbulence, and make predictions that stay coherent over time**. The result is a model that understands enough underlying structure to perform well across many domains.
80
93
81
94
---
82
95
83
-
#### Walrus in Action
84
-
85
-
<palign="center">
86
-
<imgsrc="/images/blog/walrus/Walrus_3d_examples.png"alt="Forecasts on complex 3D systems."width="95%"style="mix-blend-mode: darken;">
87
-
</p>
88
-
89
-
96
+
## Walrus in Action
97
+
<br/>
90
98
Starting from a Walrus checkpoint can speed up learning for the emulation of 2D and 3D physics across an unprecedented number of equations, boundary conditions, physical parameterizations, resolutions, and aspect ratios, offering higher accuracy on downstream tasks than prior foundation models.
91
-
99
+
<br/>
92
100
<palign="center">
93
101
<imgsrc="/images/blog/walrus/walrus_downstream_2d.png"alt="Comparisons on downstream 2D challenges"width="95%"style="mix-blend-mode: darken;">
94
-
</p>
102
+
</p>
103
+
<br/>
95
104
96
-
And this is just one of the things we demonstrate in the Walrus release. Check out the paper for more detailed experiments, or check out more rollout videos [here](https://youtube.com/playlist?list=PLqs9qkDO7oREx4_kus5671l7G-x64RXGn&si=ruyWrGjA1HDGJ8aV).
105
+
Check out the paper for more experiments, baselines, and ideas, or check out more rollout videos [here](https://youtube.com/playlist?list=PLqs9qkDO7oREx4_kus5671l7G-x64RXGn&si=ruyWrGjA1HDGJ8aV).
97
106
98
107
---
99
108
100
-
####Into the Future
101
-
109
+
### Into the Future
110
+
<br/>
102
111
The path to fully validated, production-ready machine-learned simulators will require further research, careful testing, and deeper integration with traditional methods. But Walrus shows that the foundational ideas work. It suggests a future where simulation is faster, more flexible, and more universally accessible, accelerating research across disciplines that depend on understanding the physical world.
103
112
104
113
---
105
114
106
-
#### Open Source Release & Getting Started
107
-
108
-
AION-1 is free and easy to use. Everything you need to run the model:
109
-
115
+
### Open Source Resources
116
+
<br/>
117
+
Walrus is entirely open - model and training code. You can get started here:
@@ -119,7 +127,6 @@ These types of projects wouldn't be possible without the generous support of the
119
127
division of the Flatiron Institute, a division of the Simons Foundation and from the National AI Research Resource Pilot, including support from NVIDIA
120
128
and NVIDIA’s DGX Cloud product which includes the NVIDIA AI Enterprise Software Platform.
121
129
122
-
Multi-walrus image licensed from Getty Images via Unsplash+.
123
-
Walrus in water by <ahref="https://unsplash.com/@brewbottle?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Bob Brewer</a> on <ahref="https://unsplash.com/photos/two-walins-playing-in-the-water-at-the-beach-gbxL20LPg84?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>.
130
+
Walrus title splash licensed from Getty Images via Unsplash+.
0 commit comments