High-Quality Audio-Driven Facial Animation with NVIDIA Audio2Face-3D SDK and ACE Service

Maya-ACE is a powerful Maya plugin that brings NVIDIA's Audio2Face-3D technology directly to your creative workflow. It offers two complementary solutions: local GPU processing via the Audio2Face-3D SDK for maximum performance and control, and cloud-based processing through the Audio2Face-3D NIM for scalability.

Both workflows are fully integrated and optimized within Maya-ACE. Use the local SDK for instant feedback, complete privacy, and interactive parameter tuning on your workstation. Switch to the cloud service when you need distributed processing or enterprise-scale deployment. This flexibility ensures optimal workflow for any project size.

Regardless of your chosen method, Maya-ACE delivers consistent, high-quality facial animations with the same parameters and professional results.

This repository provides everything needed for professional facial animation: a feature-rich Maya plugin, Audio2Face-3D SDK integration, service connectivity via gRPC, comprehensive test assets, and production-ready sample scenes.

The project is released under the MIT license, granting you freedom to use and modify the code, with the responsibility to ensure its appropriate use. Please note that NVIDIA assumes no liability for any issues arising from the use of this software.

Generate professional facial animations using NVIDIA's advanced AI models directly on your workstation. No internet connection or external services required.

Create animations instantly with TensorRT-optimized performance. Adjust parameters in interactive mode for immediate visual feedback and precise artistic control.

Compatible with any audio input and character design. From photorealistic humans to stylized characters, the flexible architecture adapts to your creative vision.

Capture every detail with high-resolution geometry output and full tongue articulation. Get comprehensive facial performance data for professional productions.

Integrate custom AI models and proprietary blendshape sets. The open architecture empowers technical artists to tailor the pipeline to specific production requirements.

Boost productivity with an intuitive Maya-native UI designed for artists. Full source code access enables developers to customize and integrate into existing pipelines.

Maya-ACE seamlessly integrates with the Audio2Face-3D NIM for cloud-based processing and enterprise scalability. Access the service through NVIDIA's cloud infrastructure or deploy your own private instance while maintaining the same interface and parameters as local processing. This unified experience makes Maya-ACE ideal for both individual artists and large studios.

Maya-ACE is part of NVIDIA ACE, a comprehensive suite of real-time AI solutions for interactive avatars and digital human applications. Extend your pipeline with Unreal Engine integration for real-time game characters, or build interactive customer service avatars using Tokkio. Animation parameters, pose mappings, and Audio2Face settings transfer seamlessly between platforms for consistent results.

Native integration with Maya's animation system ensures compatibility with blendshape nodes and standard Maya features. Works with any character rig in your scene.

Full source code access empowers studios to build custom pipelines or develop clients for other platforms, ensuring complete flexibility for any production environment.

• Requirements
  • Audio2Face-3D Local Inference Workflow
  • Audio2Face-3D NIM Service Workflow
  • Build Environment for Developers
• Getting Started
  • Prerequisites
    • Install the Plugin
    • Download the Sample Project
    • Download AI Models
    • Build TRT Engines
    • Prepare Access to the Service
    • Environment Variables
    • Audio File Requirements
  • Concepts
    • Architecture Overview
    • Workflow Comparison: Local Inference vs NIM
    • Model Selection Guide
    • Model Outputs
  • Quickstarts
    • Sample Scene with Audio2Face-3D Local Inference
    • Sample Scene with Audio2Face-3D Service
• User Guides
  • Setup Instructions
    • Setting Up A2F Animation Player on Blendshapes
    • Setting Up A2F Animation Player for the Geometry Output
    • Setting Up ACE Animation Player on Blendshapes
  • Best Practices
    • Parameter Tuning Guide
    • Saving Animation through Bake Animation
    • Connecting Custom Blendshapes
  • Troubleshooting Guides
• Samples Files and Production Examples
  • Sample Maya Project
    • Models
    • Scenes
    • Sound
    • Face Assets
  • Sample Data
  • Reallusion High-quality 3D Character Example
  • MetaHuman Mapping and Batch Processing
• Developer Guides
  • Setting up Development Environment
    • CUDA Toolkit
    • TensorRT
    • Audio2Face-3D SDK
    • Maya Devkit
  • Building Maya ACE Plugin
    • Build Commands
    • Debug Build
    • Output Files and Packaging
    • Testing
  • ACE gRPC C++ Library
  • Troubleshooting during Build
• User Interface
  • Compute Status - Timeslider Color
  • Attribute Editor
  • Main Menu
  • TRT Manager Window