Google’s Sima 2 Agent Uses Gemini To Reason And Act In Virtual Worlds

Google DeepMind shared connected Thursday a investigation preview of SIMA 2, nan adjacent procreation of its generalist AI supplier that integrates nan connection and reasoning powers of Gemini, Google’s ample connection model, to move beyond simply pursuing instructions to knowing and interacting pinch its environment. 

Like galore of DeepMind’s projects, including AlphaFold, nan first type of SIMA was trained connected hundreds of hours of video crippled information to study really to play aggregate 3D games for illustration a human, moreover immoderate games it wasn’t trained on. SIMA 1, unveiled successful March 2024, could travel basal instructions crossed a wide scope of virtual environments, but it only had a 31% occurrence complaint for completing analyzable tasks, compared to 71% for humans.   

“SIMA 2 is simply a measurement alteration and betterment successful capabilities complete SIMA 1,” Joe Marino, elder investigation intelligence astatine DeepMind, said successful a property briefing. “It’s a much wide agent. It tin complete analyzable tasks successful antecedently unseen environments. And it’s a self-improving agent. So it tin really self-improve based connected its ain experience, which is simply a measurement towards much general-purpose robots and AGI systems much generally.”

DeepMind says SIMA 2 doubles nan capacity of SIMA 1.Image Credits:Google DeepMind

SIMA 2 is powered by nan Gemini 2.5 flash-lite model, and AGI refers to artificial wide intelligence, which DeepMind defines arsenic a strategy tin of a wide scope of intelligence tasks pinch nan expertise to study caller skills and generalize knowledge crossed different areas. 

Working pinch alleged “embodied agents” is important to generalized intelligence, DeepMind’s researchers say. Marino explained that an embodied supplier interacts pinch a beingness aliases virtual world via a assemblage – watching inputs and taking actions overmuch for illustration a robot aliases quality would – whereas a non-embodied supplier mightiness interact pinch your calendar, return notes, aliases execute code. 

Jane Wang, a investigation intelligence astatine DeepMind pinch a inheritance successful neuroscience, told TechCrunch that SIMA 2 goes acold beyond gameplay. 

“We’re asking it to really understand what’s happening, understand what nan personification is asking it to do, and past beryllium capable to respond successful a common-sense measurement that’s really rather difficult,” Wang said. 

By integrating Gemini, SIMA 2 doubled its predecessor’s performance, uniting Gemini’s precocious connection and reasoning abilities pinch nan embodied skills developed done training.

Marino demoed SIMA 2 successful No Man’s Sky, wherever nan supplier described its surroundings – a rocky satellite aboveground – and wished its adjacent steps by recognizing and interacting pinch a distress beacon. SIMA 2 besides uses Gemini to logic internally. In different game, erstwhile asked to locomotion to nan location that’s nan colour of a ripe tomato, nan supplier showed its reasoning – ripe tomatoes are red, truthful I should spell to nan reddish location – past recovered and approached it.

Being Gemini-powered besides intends SIMA 2 follows instructions based connected emojis: “You instruct it 🪓🌲, and it’ll spell chop down a tree,” Marino said. 

Marino besides demonstrated really SIMA 2 tin navigate recently generated photorealistic worlds produced by Genie, DeepMind’s world model, correctly identifying and interacting pinch objects for illustration benches, trees, and butterflies. 

DeepMind says SIMA 2 is simply a self-improving agent.Image Credits:Google DeepMind

Gemini besides enables self-improvement without overmuch quality data, Marino added. Where SIMA 1 was trained wholly connected quality gameplay, SIMA 2 uses it arsenic a baseline to supply a beardown first model. When nan squad puts nan supplier into a caller environment, it asks different Gemini exemplary to create caller tasks and a abstracted reward exemplary to people nan agent’s attempts. Using these self-generated experiences arsenic training data, nan supplier learns from its ain mistakes and gradually performs better, fundamentally school itself caller behaviors done proceedings and correction arsenic a quality would, guided by AI-based feedback alternatively of humans.

DeepMind sees SIMA 2 arsenic a measurement toward unlocking much general-purpose robots.

“If we deliberation of what a strategy needs to do to execute tasks successful nan existent world, for illustration a robot, I deliberation location are 2 components of it,” Frederic Besse, elder unit investigation technologist astatine DeepMind, said during a property briefing. “First, location is simply a high-level knowing of nan existent world and what needs to beryllium done, arsenic good arsenic immoderate reasoning.”

If you inquire a humanoid robot successful your location to spell cheque really galore cans of beans you person successful nan cupboard, nan strategy needs to understand each of nan different concepts – what beans are, what a cupboard is – and navigate to that location. Besse says SIMA 2 touches much connected that high-level behaviour than it does connected lower-level actions, which he refers to arsenic controlling things for illustration beingness joints and wheels.

The squad declined to stock a circumstantial timeline for implementing SIMA 2 successful beingness robotics systems. Besse told TechCrunch that DeepMind’s precocious unveiled robotics instauration models – which tin besides logic astir nan beingness world and create multi-step plans to complete a ngo – were trained otherwise and separately from SIMA. 

While there’s besides nary timeline for releasing much than a preview of SIMA 2, Wang told TechCrunch nan extremity is to show nan world what DeepMind has been moving connected and spot what kinds of collaborations and imaginable uses are possible.