Life in Simulation
Researchers in the Stanford Physical and Spatial Intelligence Lab are using the computational power of Marlowe, Stanford’s GPU cluster, to create lifelike 3D simulated worlds. This work could reshape everyday life, everything from education to entertainment.
The lab is led by Stanford Associate Professor of Electrical Engineering Gordon Wetzstein, a 3D vision researcher working at the forefront of computational imagery. Some of his notable projects include designing lightweight glasses that project lifelike holographic imagery intermixed with the user’s actual surroundings and developing new ways to create photorealistic moving imagery from pure data. His goal is to push beyond the boundaries of what today’s cameras and screens can do.
With the arrival of Marlowe, Stanford’s new GPU-based supercomputer, Wetzstein’s work has expanded toward something even grander. “We’re creating world models,” Wetzstein said, explaining how he is using AI to simulate reality from data alone. He says such work will transform gaming and entertainment, of course, but also training in complex or dangerous fields like surgery and space flight. Someday, he says, human students and robots will be able to effectively learn in safe, entirely realistic simulated worlds before ever setting foot in the real one.
Worlds from data alone: the lab's model holds each scene as a global point cloud — its spatial long-term memory — and, together with working and episodic memories, uses it to condition every new frame it generates from a text prompt. Figure from the lab's paper “Video World Models with Long-term Spatial Memory”, courtesy of the Stanford Physical and Spatial Intelligence Lab. Click the figure to enlarge it.
Deluge of Data
The amount of data it takes to achieve realistic simulated environments is overwhelming. Everything from physical objects, rules of motion tied to the laws of physics, the trajectory of light, color, soundscapes, and more must be anticipated, accounted for, calculated and assembled into grids of millions of pixels going by at least 90 frames per second.
Traditional simulators struggle with what roboticists call the “sim-to-real gap.” As Wetzstein explains, “The real world is a messy and unpredictable place. A robot trained on an orderly simulated factory floor may fail in the real world when the lighting changes, smoke or dust wafts by, or an unexpected object appears in view out of nowhere.” This is where Marlowe’s computing power has proved transformational for Wetzstein’s team.
The computation grows even more complex when training two robots to collaborate in a simulation. For the first time in Wetzstein’s career, Marlowe is putting such levels of computation within reach of his academic lab. “Without Marlowe, we couldn’t do anything,” he said. “We would have to forgo the research or leave it to the big companies.”
Divide and Conquer
To tackle this massive endeavor, Wetzstein breaks down the problem of creating simulated life into more manageable components. For example, Wetzstein and team are working on a form of long-term memory to ensure AI-generated worlds remain consistent and true over time.
That memory at work: the model generates a scene frame by frame while three memory stores run alongside it — a spatial point cloud, a rolling window of recent frames, and a set of keyframes — and feed back into what it draws next, which is what keeps the world consistent as the viewpoint moves through it.
The everyday impact of such advances for most audiences is still far off, but at least they are imaginable, Wetzstein says. He predicts virtual reality of the future will be true to life: domestic robots will practice in simulated homes that match those where they will be deployed; training simulators will adapt on the fly as trainees move about and make decisions freely; games and entertainment will flow responsively, guided by the actions of the players.
Computational might, like that which Marlowe provides, will be important to the future of academia itself. While currently compute power is largely accessible only to a handful of large companies, Wetzstein predicts a future world of ideas where methods can be openly tested, scrutinized, and published – where imagination and talent are the only constraints, not compute power.
“The simulator of the future,” Wetzstein concluded, “will change how we train machines, how we learn new skills ourselves, and how we experience the real world and Marlowe makes it possible.”