# TAO [NVIDIA TAO (Train Adapt Optimize) Toolkit](https://developer.nvidia.com/tao-toolkit) is a framework for customizing vision foundation models to achieve high accuracy and performance with fine-tuning microservices. It provides transfer learning capability to adapt popular neural network architectures and backbones to your data, allowing you to train, fine-tune, prune, quantize and export highly optimized and accurate AI models for edge deployment. You can use TAO Toolkit to distill knowledge from large foundation models like C-RADIOv2 and NVDINOv2 to smaller compute friendly models for the edge. The purpose built pre-trained models accelerate the AI training process and reduce costs associated with large scale data collection, labeling, and training models from scratch. Transfer learning with pre-trained models can be used for AI applications in smart cities, retail, healthcare, industrial inspection and more. TAO supports training for CV and 3D Point cloud modalities. TAO packages a collection of containers, python wheels, models and helm chart. AI training tasks run on PyTorch depending upon the entrypoint for the model. For deployment, TAO models can be deployed to [DeepStream](https://developer.nvidia.com/deepstream-sdk) for video analytics applications, or [Triton](https://developer.nvidia.com/nvidia-triton-inference-server) for inference serving use cases. ## License Use of this software is governed by the [NVIDIA Software License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and [Product-Specific Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/) for NVIDIA AI Products. Your use of models with this software is subject to the terms accompanying the models. ## TAO 6.25.9 FTMS TAO 6.25.9 FTMS includes the following assets: 1. 3 containers 2. A [getting started resource](https://github.com/NVIDIA/tao_tutorials) that contains tutorial notebooks 3. NVIDIA TAO launcher wheel (hosted on [PyPI](https://pypi.org/project/nvidia-tao)) 4. TAO Helm [Chart](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/helm-charts/tao-toolkit-api) 5. [C-RADIOv2 foundation models](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/cradiov2) 6. [ConvNext series of Foundation models](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/convnextv2) 7. Purpose built model - [TrafficCamNet Transformer Lite](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/trafficcamnet_transformer_lite) for object detection in traffic scenes. 8. State of the art commercial models for depth estimation from monocular images and stereo image pairs: 1. Monocular depth estimation via [NvDepthAnythingv2](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/nvdepthanythingv2) 2. Stereo depth estimation via [C-FoundationStereo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/foundationstereo) 9. Purpose built model - [Sparse4D](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/sparse4d_rn101) for multi-camera 3D object detection and tracking. ### TAO Containers All containers needed to run TAO can be pulled from this location. See the list below for all available containers in this registry. | **TAO Container Type** | **container_name:tag** | **What's it used for?** | |:-|:-|:-| | TAO PyTorch container | nvcr.io/nvidia/tao/tao-toolkit:6.25.9-pyt | Finetuning workflows in PyTorch | | TAO Deploy container | nvcr.io/nvidia/tao/tao-toolkit:6.25.9-deploy | TensorRT inference workflows in Deploy | | TAO Data Services container | nvcr.io/nvidia/tao/tao-toolkit:6.25.9-data-services | Dataset augmentation, autolabelling and analysis workflows | ## How to run TAO 6.25.9 FTMS? To get started with TAO, download the tutorial notebook using the NGC CLI and run the following commands. > Note: The instructions to install the NGC CLI are detailed [here](https://docs.ngc.nvidia.com/cli/cmd.html). ```sh git clone https://github.com/NVIDIA/tao_tutorials.git git checkout release/6.25.9 ``` You can run the kick-start the notebook by running ```sh python3 -m pip install nvidia-tao --upgrade python3 -m pip install jupyter jupyter notebook --ip 0.0.0.0 --port --allow-root ``` ### Pre-trained Models The TAO 6.25.9 FTMS package refers several pre-trained models released as part of NGC. | **Model Name** | **Pulled** | **Use Case** | |---|:-:|---| | [CRADIOv2](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/cradiov2)| Manual | Multi-teacher distilled foundation model generating rich visual embeddings | | [ConvNextv2](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/convnextv2)| Manual | FC-MAE trained foundation model generating rich visual embeddings for CNNs | | [TrafficCamNet Transformer Lite](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/trafficcamnet_transformer_lite)| Manual | Object detection network for detecting 4 class objects in traffic scenes | | [NvDepthAnythingv2](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/nvdepthanythingv2) | Manual | Depth estimation model to generate relative depth maps from monocular images | | [C-FoundationStereo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/foundationstereo) | Manual | Depth estimation model to generate relative disparity maps from stereo image pairs| | [Sparse4D](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/sparse4d_rn101) | Manual | Multi-camera 3D object detection and tracking | DeiT-base, EfficientNet, and DeiT-small are pulled automatically through the timm API in the training code. C-RADIOv2, InceptionNet and EfficientNet-embeddings need to be pulled manually. The tutorial notebooks outline the instructions and steps to pull these models. ## Technical blogs - [Access the latest in Vision AI development workflows with NVIDIA TAO Toolkit 5.0](https://developer.nvidia.com/blog/access-the-latest-in-vision-ai-model-development-workflows-with-nvidia-tao-toolkit-5-0/) - [Improve accuracy and robustness of vision ai models with vision transformers and NVIDIA TAO](https://developer.nvidia.com/blog/improve-accuracy-and-robustness-of-vision-ai-apps-with-vision-transformers-and-nvidia-tao/) - [Train like a ‘pro’ without being an AI expert using TAO AutoML](https://developer.nvidia.com/blog/training-like-an-ai-pro-using-tao-automl/) - [Create Custom AI models using NVIDIA TAO Toolkit with Azure Machine Learning ](https://developer.nvidia.com/blog/creating-custom-ai-models-using-nvidia-tao-toolkit-with-azure-machine-learning/) - [Developing and Deploying AI-powered Robots with NVIDIA Isaac Sim and NVIDIA TAO](https://developer.nvidia.com/blog/developing-and-deploying-ai-powered-robots-with-nvidia-isaac-sim-and-nvidia-tao/) ## Suggested reading - More information about TAO Toolkit and pre-trained models can be found at the [NVIDIA Developer Zone](https://developer.nvidia.com/tao-toolkit) - [TAO documentation](https://docs.nvidia.com/tao/tao-toolkit/index.html) - Read the [TAO getting Started](https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_quick_start_guide.html) guide and [release notes](https://docs.nvidia.com/tao/tao-toolkit/text/release_notes.html). - If you have any questions or feedback, please refer to the discussions on [TAO Toolkit Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/tao-toolkit/17) - Deploy your models for video analytics application using DeepStream. Learn more about [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk) - Deploy your models in [Riva](https://developer.nvidia.com/riva) for ConvAI use case. ## Ethical Considerations NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended. ## Security Vulnerabilities in Open Source Packages Please review the [Security Scanning](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/containers/tao-toolkit/security) section to view the latest security scan results. For certain open-source vulnerabilities listed in the scan results, NVIDIA provides a response in the form of a Vulnerability Exploitability eXchange (VEX) document. The VEX information can be reviewed and downloaded from the [Security Scanning](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/containers/tao-toolkit/security) section.