3D Object Generation Blueprint

Description

The 3D Object Generation Blueprint is an end-to-end generative AI workflow that allows users to prototype 3D scenes quickly by simply describing the scene. The Blueprint takes a user’s 3D scene idea, generates object recommendations, associated prompts and previews using a Llama 3.1 8B LLM and NVIDIA SANA, and ready-to-use 3D objects with Microsoft TRELLIS.

This blueprint supports the following NVIDIA GPUs: RTX 5090, RTX 5080, RTX 4090, RTX 4080, RTX 6000 Ada. We're planning to add wider GPU support in the near future. We recommend at least 48 GB of system RAM.

Features

Chat interface for scene planning
AI-assisted object and prompt generation
Automatic 3D asset generation from text prompts
Blender import functionality for generated assets
VRAM management with model termination

Installation

Prerequisites

The NIM Prerequisite Installer requires Microsoft User Account Control (UAC) to be enabled. UAC is enabled by default for Windows, but if it has been disabled, it must be enabled to ensure successful installation of the NIM Prerequisite Installer. More information on Microsoft UAC can found HERE

Use winget to install Miniconda:

winget install miniconda3

Download the NIM Prerequisite Installer, and run the NIMSetup.exe file, and follow the instructions in the setup dialogs. This will install the necessary system components to work with NVIDIA NIMs on your system.

You will need to reboot your computer to complete the installation.

Git is required and should be installed using winget from a command prompt::

winget install --id Git.Git

Install Git LFS (large file system) support

winget install --id=GitHub.GitLFS  -e

This blueprint requires the installation of Blender. The blueprint has been tested with the Blender 4.27 LTS (Long Term Support) build.
https://www.blender.org/download/release/Blender4.2/blender-4.2.7-windows-x64.msi

Blender 4.27 can also be installed using winget from a command prompt:

winget install --id 9NW1B444LDLW

Initilize Conda

From the Windows Start button, open a Anaconda (Miniconda) Prompt
You should see the command prompt with prompt prefixed with the (base) conda environment name
Initialize Conda for your system

conda init

Close the command prompt

Installation Steps

Open a new command prompt
Clone this repository:

git clone https://github.com/NVIDIA-AI-Blueprints/3d-object-generation.git

Run the installer script:

cd 3d-object-generation
.\install.bat

The installation process will:

Create a Conda virtual environment
Install all required dependencies
Set up necessary configurations

After successful installation, you'll see:

Installation completed successfully
Press any key to continue . . .

Usage - Manual Steps

Starting the Application

In a new terminal window, start the main application:

conda activate trellis
cd 3d-object-generation

python app.py

Open your browser to the URL shown in the terminal (typically http://127.0.0.1:7860/)

💡 Recommended: For the best experience, use the light theme by accessing the application with: http://127.0.0.1:7860/?__theme=light

⚠️ Important Note: Browser refresh is not supported and may cause the application to crash. In that case, please restart the application instead.

Managing the Application

To terminate the application and free VRAM: Open a command prompt at the 3d-object-generation install directory

conda activate trellis
cd 3d-object-generation
python terminator.py

This will:

Gracefully terminate the Gradio application
Free up GPU memory
Allow you to proceed with other operations which may require VRAM (e.g., Blender)

Usage - Blender Add On

The 3D Object Generation Blender add-on can automatically manage required services without the need to manually start or stop these services outside of Blender.

Initial Setup Step

Open Blender
Open Edit >> Preferences >> Add-Ons 3.
Enable 3D Object Generation and Asset Importer by checking the boxes next to the add on names.
Open the 3D Object Generation add on preferences and set the Blueprint Base folder to the 3d-object-generation local repository directory (This setting should be automatically set, but may be manually set if desired).

Normal Usage

In the 3D layout view look for the Add On tabs on the right edge of the viewport, press N if they are not visible
Note: It is recommended to open a system console viewer to monitor the services and any information or errors that may be output. a. Blender Menu >> Window >> Toggle System Console
Click the Start Start Services button to start the LLM agent, and the Trellis 3D services. (It may take up to 3 minutes for all services to fully load and start)
Once all services have successfully started, the service will indicate: READY and the Open 3D Object Generation UI button will become available
Click the Open 3D Object Generation UI button to launch the 3D Object Generation interface.
Clicking the Services Started .. Click to Terminate button will shut down the LLM and Trellis services and release system resources being utilized.

Using the Interface

Once the application is running, you can:

Scene Planning:
- Describe your desired scene in natural language
Asset Generation:
- The LLM will automatically create prompts for suggested items which will be sent to the 2D image generator
- Each image contains additional controls
  - Refresh - Generate a new image based on the existing prompt.
  - Edit - Edit the prompt an generate a new image.
  - Delete - Remove the image from the gallery display.
  - Generate 3D - Generate a 3D object from the image.
    - The color of the Generate 3D button indicates the status of 3D generation for that object
    - Object has not been queued for 3D generation.
    - Object has been queued for 3D generation, but object generation has not completed.
    - 3D model has been generated for this object.
    - Object has been flagged by guardrails as potentially inappropriate, 3D object will not be generated.

- Convert all images to 3D Objects (Delete unwanted images before converting to 3D)

NOTE: Image to 3D Object processing takes up to 45 seconds per object on a RTX 5090, when using the Convert All image option this time will be a multiple of the number of objects being converted, using the Convert All option may take a significant amount of time. The UI will not be updated until all objects have been converted.

Save Objects:
- The Export Objects to File allows saving the generated objects to a folder.
Blender Integration:
- Import generated assets directly into Blender
- Use the Asset Importer add-on and select the desired scene folder, and click Import assets
- Assets are imported and the asset tag is applied, saving the scene to the %userprofile%\Documents\Blender\assets folder will add the imported objects to the Blender asset browser.
- Continue working with the assets in your 3D workflow
- Can be used with 3D Guided Gen AI BP

Manual Installation Guide for Trellis Project

This guide provides a step-by-step manual for setting up the Trellis environment on a Windows system with Conda, as a manual alternative to running the operations via the install.bat script. It covers checking prerequisites, creating a Conda environment, installing dependencies, setting environment variables, downloading models, installing Blender addons, and verifying services by starting, checking, and stopping them.

Prerequisites:

Conda must be installed and added to your system's PATH.
Required files in the current directory: requirements-torch.txt, requirements.txt, set_environment_variable.py, download_models.py, check_services.py.
Subdirectories: blender (containing NV_Trellis_Addon.py and asset_importer.py), nim_llm (containing run_llama.py and manager.py), nim_trellis (containing run_trellis.py and manager.py).
Blender installed (versions 4.2 to 5.0 supported; defaults to 4.2 if no version detected).

Run commands in a Command Prompt with administrative privileges where possible. Replace placeholders (e.g., paths) as needed for your setup.

Step 1: Check for Conda Installation

Open Command Prompt.
Run: where conda.
If Conda is not found (errorlevel ≠ 0), install Conda from the official website (e.g., Miniconda or Anaconda) and add it to PATH. Exit if not installed.

Step 2: Verify Requirements Files

Ensure requirements-torch.txt exists in the current directory. If not, obtain it and place it there.
Ensure requirements.txt exists in the current directory. If not, obtain it and place it there.
If either file is missing, stop the process and resolve the issue.

Step 3: Locate Conda Installation Path

Check common Conda installation paths:
- %USERPROFILE%\miniconda3
- %USERPROFILE%\AppData\Local\miniconda3
- %USERPROFILE%\anaconda3
Verify the presence of Scripts\conda.exe or condabin\conda.bat in one of these paths.
If not found, verify your Conda installation and adjust paths manually. Set CONDA_PATH to the valid location.

Step 4: Accept Anaconda Terms of Service (If Applicable)

Run the following commands to accept terms for Anaconda channels (note: this may be specific to certain setups; skip if conda tos accept is not recognized):
- conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
- conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
- conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/msys2
If the command fails, check Conda documentation for equivalent configuration steps (e.g., using conda config).

Step 5: Create or Verify Conda Environment

List existing environments: conda env list.
If "trellis" does not exist:
- Create it: conda create -n trellis python=3.11.9 -y.
If creation fails, troubleshoot Conda issues (e.g., network, permissions).

Step 6: Activate Conda Environment

Activate: conda activate trellis.
If activation fails, ensure the environment exists and retry.

Step 7: Update Pip and Install Build Tools

Upgrade pip and wheel: python -m pip install --upgrade pip wheel.
Install setuptools: python -m pip install setuptools==75.8.2. (Note: Version is hardcoded; use latest if needed.)
If any installation fails, check pip logs or network connectivity.

Step 8: Install Torch Requirements

Install packages from file: pip install -r requirements-torch.txt.
If fails, verify the file contents and resolve dependency conflicts.

Step 9: Install Main Requirements

Install packages from file: pip install -r requirements.txt.
If fails, verify the file contents and resolve dependency conflicts.

Step 10: Set Environment Variable

Run: python set_environment_variable.py (this sets CHAT_TO_3D_PATH).
Verify the variable is set: echo %CHAT_TO_3D_PATH%.
If the script fails, inspect set_environment_variable.py for issues.

Step 11: Download Required Models

Note your current directory: echo %cd%.
Run: python download_models.py.
If fails, check the script for download URLs and ensure internet access.
Return to the original directory if changed.

Step 12: Install Blender Addons

Verify blender folder exists with NV_Trellis_Addon.py and asset_importer.py.
Set Blender root: %APPDATA%\Blender Foundation\Blender.
If the root doesn't exist, create it: mkdir "%APPDATA%\Blender Foundation\Blender".
Check for version folders (4.2 to 5.0).
For each matching version (e.g., 4.2):
- Create addons dir if needed: mkdir "<BlenderRoot>\<version>\scripts\addons".
- Copy files: copy blender\NV_Trellis_Addon.py "<addons_dir>\NV_Trellis_Addon.py" and copy blender\asset_importer.py "<addons_dir>\asset_importer.py".
If no versions found, create default 4.2 folder and copy addons as above.
After copying, enable the addon in Blender preferences.

Step 13: Deactivate Conda Environment (Temporarily)

Run: conda deactivate.

Step 14: Start LLM and Trellis Services

Start LLM in background: Open a new Command Prompt and run conda activate trellis && python nim_llm\run_llama.py.
Wait 10 seconds.
Start Trellis in background: Open another Command Prompt and run conda activate trellis && python nim_trellis\run_trellis.py.
Wait 10 seconds.
Check running processes: tasklist /FI "IMAGENAME eq python.exe" /FO TABLE.

Step 15: Monitor Services Readiness

Reactivate environment in main prompt: conda activate trellis.
Loop to check readiness (up to 150 attempts, ~75 minutes):
- Run: python check_services.py.
- Interpret exit code:
  - 0: Both ready.
  - 1: LLM ready, Trellis not.
  - 2: Trellis ready, LLM not.
  - Other: Neither ready.
- Wait 30 seconds between checks.
Services endpoints: LLM at http://localhost:19002, Trellis at http://localhost:8000.
If timeout, check logs: nim_llm\llama_container.log and nim_trellis\trellis_container.log.

Step 16: Stop Services

Stop LLM: python -c "from nim_llm.manager import stop_container; stop_container()".
Stop Trellis: python -c "from nim_trellis.manager import stop_container; stop_container()".
Kill remaining Python processes: taskkill /f /im python.exe.
Deactivate environment: conda deactivate.

Completion

If all steps succeed without errors, the installation is complete. You can now use the Trellis environment. Troubleshoot any errors by checking console output, logs, or dependencies. For first-time setups, model downloads and service starts may take significant time (60-120 minutes).

Troubleshooting

Common issues and solutions:

Installation Issues:
- Ensure all prerequisites are installed correctly
- Check if Python is in your system PATH
- Verify Visual Studio Build Tools installation
Runtime Issues:
- Make sure both NIM and main application are running
- Check GPU memory usage
- Verify all environment variables are set correctly

Acknowledgments

TRELLIS for the 3D generation capabilities
Griptape for the agent framework
Gradio for the web interface

Name		Name	Last commit message	Last commit date
Latest commit History 91 Commits
.github		.github
blender		blender
components		components
nim_llm		nim_llm
nim_trellis		nim_trellis
services		services
static		static
.gitattributes		.gitattributes
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
app.py		app.py
check_services.py		check_services.py
config.py		config.py
download_models.py		download_models.py
install.bat		install.bat
requirements-torch.txt		requirements-torch.txt
requirements.txt		requirements.txt
set_environment_variable.py		set_environment_variable.py
start_services.bat		start_services.bat
stop_services.bat		stop_services.bat
terminator.py		terminator.py
utils.py		utils.py

License

NVIDIA-AI-Blueprints/3d-object-generation

Folders and files

Latest commit

History

Repository files navigation