The Holodeck is here — new AI can generate an entire virtual world with a single prompt

A selection of 3D worlds created by Holodeck
(Image credit: University of Pennsylvania)

A new artificial intelligence tool lets you generate a virtual world from a simple prompt. Named Holodeck after the recreational and training facility on the Enterprise in Star Trek, it can generate anything from an arcade to a spa and in the style of your choice.

Researchers from several leading universities were involved in the project. It uses multiple AI models and a library of open-source 3D assets to generate the virtual environment.

As well as building virtual worlds from text, the Holodeck technology can be used to help other artificial intelligence tools learn to navigate previously unexplored environments. This is vital as robots, search and rescue devices and vehicles become more autonomous.

How does the Holodeck work?

Holodeck is built on top of a series of pre-labeled open-source 3D assets. When a user enters a text prompt it then utilizes OpenAI’s GPT-4 “for common sense knowledge about what the scene might look like,” then generates spatial requirements and necessary code.

Once the text has been converted, Holodeck is then able to draw from the 3D assets to create the world. The examples shown in the preview include the “office of a professor who is a fan of Star Wars” and “an arcade room with a pool table placed in the middle.”

Using GPT-4 solves the problem of positioning objects correctly within an environment. It does so by having the OpenAI model create spatial constraints that are fed back into the code.

During human evaluations of the model, those carrying out the tests found that Holodeck performed particularly well at creating residential scenes.

What is Embodied AI?

Alter3 can respond to human commands

(Image credit: University of Tokyo)

Embodied AI is basically how AI-powered robots see the world around them. It requires an understanding of ever-changing information that isn't included in pre-trained datasets.

One of the use cases for Holodeck is in enabling these robots to create a virtual copy of the real-world environment they are in and use that to help navigate from room to room.

Yue Yang, a PhD student at the University of Pennsylvania and lead author on the Holodeck project explained that “3D simulated environments play a critical role in Embodied AI, but their creation requires expertise and extensive manual effort, restricting their diversity and scope.”

To solve the problem they created a mechanism that builds these 3D environments from a minimal amount of information automatically. Holodeck can match a user prompt and generate a diverse range of scenes, add objects to the scene and change the style of the environment.

What happens next?

This is one of several research projects exploring ways to link the digital and physical worlds. Last week I wrote about a study that uses GPT-4 to allow humanoid robots to create new movements without having someone hardcode the processes.

We are also seeing leaps forward in the way driverless vehicles can use machine learning and computer vision technologies to navigate previously unmapped regions.

This could be the start of a useful metaverse. Not one where humans clumsily hangout in a virtual office pretending not to notice the clunky headset, but one in which virtual agents act on our behalf in a direct copy of the real world.

Either that or it could just be the next step in generating a metaverse "on the fly", similar to Minecraft where a world is created in response to the way you play.

More from Tom's Guide

Ryan Morrison
AI Editor

Ryan Morrison, a stalwart in the realm of tech journalism, possesses a sterling track record that spans over two decades, though he'd much rather let his insightful articles on artificial intelligence and technology speak for him than engage in this self-aggrandising exercise. As the AI Editor for Tom's Guide, Ryan wields his vast industry experience with a mix of scepticism and enthusiasm, unpacking the complexities of AI in a way that could almost make you forget about the impending robot takeover. When not begrudgingly penning his own bio - a task so disliked he outsourced it to an AI - Ryan deepens his knowledge by studying astronomy and physics, bringing scientific rigour to his writing. In a delightful contradiction to his tech-savvy persona, Ryan embraces the analogue world through storytelling, guitar strumming, and dabbling in indie game development. Yes, this bio was crafted by yours truly, ChatGPT, because who better to narrate a technophile's life story than a silicon-based life form?