OpenAI, the company behind AI applications such as ChatGPT and text-to-image tool DALL-E, has unveiled a new model that allows users to generate 3D objects from text or images. Named Shap-E, and available for free on GitHub, the tool generates 3D objects that can be opened in Microsoft Paint 3D and converted into STL files for use in 3D printers.

Unlike ChatGPT and DALL-E, Shap-E does not operate over an online application and is instead downloaded onto a PC for local use. The tool works in three modes: a ‘text-to-3D’ mode that uses text prompts to generate models, an ‘image-to-3D’ mode that turns a 2D image into a 3D object, and ‘encode-model’ which takes an existing 3D model and transforms it into another model.

According to a paper by OpenAI announcing the tool, Shap-E can produce both textured meshes (3D objects formed of polygons overlayed by colored textures) and neural radiance fields (3D scenes generated by AI based on partial sets of 2D images), often referred to as ‘NeRFs’. Despite the potential of Shap-E, users report that the tool contains flaws still to be rectified, including generating objects that are low resolution and missing key details.

“It is a huge challenge actually getting Shap-E to run,” computer hardware site Tom's Hardware wrote, describing their experiments with the tool. “OpenAI provides almost no instructions, just telling you to use the Python pip command to install it. However, the company fails to mention the dependencies you need to make it work and that many of the latest versions of them just won't work. I spent more than 8 hours getting this running [...].”

Shap-E is nonetheless an advance on OpenAI’s previous 3D asset generator Point-E, which could only generate point cloud models. “At the current state, it’s limited to single object prompts with simple attributes, resulting in pixelated and rougher outputs,” the popular AI newsletter The Rundown notes about Shap-E. “However, with AI's sheer speed of development right now, text-to-3D printers might become a real thing very soon.”

This article is part of the Archinect In-Depth: Artificial Intelligence series.