- Overview
- Use Case / Problem Description
- Software Components
- Target Audience
- Repository Structure Overview
- Documentation
- Prerequisites
- Hardware Requirements
- Quickstart Guide
- Known CVEs
- License
This repository is what powers the build experience, showcasing video search and summarization agent with NVIDIA NIM microservices.
Insightful, accurate, and interactive video analytics AI agents enable a range of industries to make better decisions faster. These AI agents are given tasks through natural language and can perform complex operations like video summarization and visual question-answering, unlocking entirely new application possibilities. The NVIDIA AI Blueprint makes it easy to get started building and customizing video analytics AI agents for video search and summarization — all powered by generative AI, vision language models (VLMs) like Cosmos Nemotron VLMs, large language models (LLMs) like Llama Nemotron LLMs, NVIDIA NeMo Retriever, and NVIDIA NIM.
The NVIDIA AI Blueprint for Video Search and Summarization addresses the challenge of efficiently analyzing and summarizing large volumes of video data. This can be used to create vision AI agents, that can be applied to a multitude of use cases such as monitoring smart spaces, warehouse automation, and SOP validation. This is important where quick and accurate video analysis can lead to better decision-making and enhanced operational efficiency.
-
NIM microservices: Here are models used in this blueprint:
-
Ingestion Pipeline:
The process involves decoding video segments (chunks) generated by the stream handler, selecting frames, and using a vision-language model (VLM) along with a caption prompt to generate detailed captions for each chunk. A computer vision pipeline enhances video analysis by providing detailed metadata on objects. In parallel, the audio is extracted and a transcription is generated. These dense captions, along with audio transcripts and CV metadata are then indexed into vector and graph databases for use in the Context-Aware Retrieval-Augmented Generation workflow.
-
CA-RAG module:
The Context-Aware Retrieval-Augmented Generation (CA-RAG) module leverages both Vector RAG and Graph-RAG as primary sources for video understanding. This module is utilized in key features such as summarization, Q&A, and sending alerts. During the Q&A workflow, the CA-RAG module extracts relevant context from the vector database and graph database to enhance temporal reasoning, anomaly detection, multi-hop reasoning, and scalability. This approach offers deeper contextual understanding and efficient management of extensive video data. Additionally, the context manager effectively maintains its working context by efficiently using both short-term memory, such as chat history, and long-term memory resources like vector and graph databases, as needed.
This blueprint is designed for ease of setup with extensive configuration options, requiring technical expertise. It is intended for:
-
Video Analysts and IT Engineers: Professionals focused on analyzing video data and ensuring efficient processing and summarization. The blueprint offers 1-click deployment steps, easy-to-manage configurations, and plug-and-play models, making it accessible for early developers.
-
GenAI Developers / Machine Learning Engineers: Experts who need to customize the blueprint for specific use cases. This includes modifying the RAG pipeline for unique datasets and fine-tuning LLMs as needed. For advanced users, the blueprint provides detailed configuration options and custom deployment possibilities, enabling extensive customization and optimization.
deploy/
: Contains scripts for docker compose and helm chart deployment, along with notebook for Brev launchable deployment.src/
: Source code for the video search and summarization agent.examples/
: Training notebooks for using VSS and usecase examples.
For detailed instructions and additional information about this blueprint, please refer to the official documentation.
- NVIDIA AI Enterprise developer licence required to local host NVIDIA NIM.
- API catalog keys:
- NVIDIA API catalog or NGC (steps to generate key)
The platform requirement can vary depending on the configuration and deployment topology used for VSS and dependencies like VLM, LLM, etc. For a list of validated GPU topologies and what configuration to use, see the supported platforms.
Deployment Type | VLM | LLM | Embedding (llama-3.2-nv-embedqa-1b-v2) | Reranker (llama-3.2-nv-rerankqa-1b-v2) | Minimum GPU Requirement |
---|---|---|---|---|---|
Local deployment (Default topology) | Local (Cosmos Reason1 7B) | Local (Llama 3.1 70B) | Local | Local | 8xB200, 8xH200, 8xH100, 8xA100 (80GB), 8xL40S, 8xRTX PRO 6000 Blackwell |
Local deployment (Reduced Compute) | Local (Cosmos Reason1 7B) | Local (Llama 3.1 70B) | Local | Local | 4xB200, 4xH200, 4xH100, 4xA100 (80GB), 6xL40S, 4xRTX PRO 6000 Blackwell |
Local deployment (Single GPU) | Local (Cosmos Reason1 7B) | Local (Llama 3.1 8b low mem mode) | Local | Local | 1xB200, 1xH200, 1xH100, 1xA100 (80GB), 1xRTX PRO 6000 Blackwell, DGX Spark |
Local VLM deployment | Local(Cosmos Reason1 7B) | Remote | Remote | Remote | 1xB200, 1xH200, 1xH100, 2xA100 (80GB), 1xL40S, 1xRTX PRO 6000 Blackwell, Jetson Thor, DGX Spark |
Complete remote deployment | Remote | Remote | Remote | Remote | Minimum 8GB VRAM GPU, Jetson Thor, DGX Spark |
Ideal for: Quickly getting started with your own videos without worrying about hardware and software requirements.
Follow the steps from the documentation and notebook in deploy directory to complete all pre-requisites and deploy the blueprint using Brev Launchable in an 8xL040s Crusoe instance.
- deploy/1_Deploy_VSS_docker_Crusoe.ipynb: This notebook is tailored specifically for the Crusoe CSP which uses Ephemeral storage.
Ideal for: Development phase where you need to run VSS locally, test different models, and experiment with various deployment configurations. This method offers greater flexibility for debugging each component.
For custom VSS deployments through Docker Compose, multiple samples are provided to show different combinations of remote and local model deployments. The /deploy/docker
directory contains a README with all the details. Link to README
- Ubuntu 22.04
- NVIDIA driver 580.65.06 (Recommended minimum version)
- CUDA 13.0+ (CUDA driver installed with NVIDIA driver)
- NVIDIA Container Toolkit 1.13.5+
- Docker 27.5.1+
- Docker Compose 2.32.4
Please refer to Prerequisites section here for more information.
Please refer to NVIDIA Jetson Thor Setup Instructions.
Please refer to NVIDIA DGX Spark Setup Instructions.
Ideal for: Production deployments that need to integrate with other systems. Helm offers advantages such as easy upgrades, rollbacks, and management of complex deployments.
The /deploy/helm/
directory contains a nvidia-blueprint-vss-2.4.0.tgz
file which can be used to spin up VSS. Refer to the documentation here for detailed instructions.
- Ubuntu 22.04
- NVIDIA driver 580.65.06 (Recommended minimum version)
- CUDA 13.0+ (CUDA driver installed with NVIDIA driver)
- Kubernetes v1.31.2
- NVIDIA GPU Operator v23.9 (Recommended minimum version)
- Helm v3.x
NOTE: Helm deployments are supported only for x86 platforms.
VSS Engine 2.4.0 Container has the following known CVEs:
CVE | Description |
---|---|
CVE-2024-8966 | This impacts gradio <= 5.22.0 python package, This impacts the file upload functionality of Gradio UI where an attacker can cause Denial-of-Service (DoS) attack by appending a large number of characters to the end of a multipart boundary. This does not affect VSS since the underlying root cause is already fixed by having a newer version 0.0.18 of python-multipart which does not have this vulnerability. |
CVE-2025-4565 | This impacts protobuf < 4.25.8 python package, This impacts parsing of untrusted Protocol Buffers data containing an arbitrary number of recursive groups, recursive messages or a series of SGROUP tags leading to unbounded recursions and potential Denial-of-Service when protobuf pure-Python backend is used. This does not affect VSS since python backend of protobuf is not used. |
CVE-2025-3887 | This impacts GStreamer H.265 codec parser, Malicious malformed streams can cause stack overflow in H.265 codec parser causing the application to crash. Users must take care that malicious H.265 streams are not added to VSS. This can be remedied by building and installing the GStreamer1.24.2 codec parser library after applying the patch mentioned in https://gstreamer.freedesktop.org/security/sa-2025-0001.html. |
Refer to LICENSE