Body Pose Estimation

Body pose estimation is a computer vision task that involves recognizing specific points on the human body in images or videos. A model is used to infer the locations of keypoints from the source video which is then rendered by the visualizer.

Model

This application uses YOLO11 pose model from Ultralytics for body pose estimation. The model is downloaded when building the application.

Data

This application downloads a pre-recorded video from Pexels when the application is built for use with this application. Please review the license terms from Pexels.

NOTE: The user is responsible for checking if the dataset license is fit for the intended purpose.

Input

This app supports three different input options. If you have a v4l2 compatible device plugged into your machine such as a webcam, you can run this application with option 1. Otherwise you can run this application using a pre-recorded video with option 2. For more advanced use cases, option 3 shows how to publish/subscribe to a DDS video stream to for interprocess applications.

v4l2 compatible input device (default, see V4L2 Support below)
pre-recorded video (see Video Replayer Support below)
DDS video stream (see DDS Support below)

To see the list of v4l2 devices connected to your machine, install v4l-utils if it's not already installed:

sudo apt-get install v4l-utils

Then run:

v4l2-ctl --list-devices

Run Instructions

V4L2 Support

This application supports v4l2 compatible devices as input. To run this application with your v4l2 compatible device, please plug in your input device and run:

./holohub run body_pose_estimation

By default, this application expects the input device to be mounted at /dev/video0. If this is not the case, please update applications/body_pose_estimation/body_pose_estimation.yaml and set it to use the corresponding input device before running the application. You can also override the default input device on the command line by running:

./holohub run body_pose_estimation --run-args="--video_device /dev/video0"

Video Replayer Support

If you don't have a v4l2 compatible device plugged in, you may also run this application on a pre-recorded video. To launch the application using the Video Stream Replayer as the input source, run:

./holohub run body_pose_estimation --run-args="--source replayer"

or using the mode replayer defined in the metadata.json file:

./holohub run body_pose_estimation replayer

DDS Support

This application supports using a DDS video stream as the input as well as publishing the output video stream back to DDS. To enable DDS, the application must first be built with the DDS operators enabled. Only the subscriber or publisher operators need to be enabled for the sake of input or output video streams, respectively, but to enable both use the following:

./holohub build --local body_pose_estimation --build-with "dds_video_subscriber;dds_video_publisher"

Note that building these operators requires RTI Connext be installed. See the DDS Operator Documentation for more information on how to build the operators. If using a development container, see the additional instructions below.

To use a DDS video stream as the input to the application, use the -s=dds argument when running the application:

./holohub run --local body_pose_estimation --run-args="-s=dds"

or using the mode dds defined in the metadata.json file:

./holohub run body_pose_estimation dds --local

To publish the output result to DDS, edit the body_pose_estimation.yaml configuration file so that the dds_publisher enable option is true:

dds_publisher:
  enable: true

Note that the default DDS video stream IDs use by the application are 0 for the input and 1 for the output. These can be changed using the stream_id settings in the dds_source and dds_publisher sections of the configuration file, respectively.

To produce the DDS input stream or to view the output stream generated by this application, the dds_video application can be used. For example, the following will use the dds_video application to capture video from the default V4L2 device and publish it to DDS so that it can be received as input by this application:

./holohub run --local dds_video --run-args="-p -i 0"

And the following will use the dds_video application to receive and render the output published by this application:

./holohub run --local dds_video --run-args="-s -i 1"

Using a Development Container with DDS Support

Installing RTI Connext into the development container is not currently supported, so enabling DDS support with this application requires RTI Connext be installed onto the host and then mounted into the container at runtime. To mount RTI Connext into the container, ensure that the NDDSHOME and CONNEXTDDS_ARCH environment variables are set (which can be done using the RTI setenv script) then use the following:

./holohub run-container body_pose_estimation --docker-opts="-v $NDDSHOME:/opt/dds -e NDDSHOME=/opt/dds -e CONNEXTDDS_ARCH=$CONNEXTDDS_ARCH"

Known Issues

There is a known issue running this application on IGX w/ iGPU and on Jetson AGX (see #500). The workaround is to update the device to avoid picking up the libnvv4l2.so library.

cd /usr/lib/aarch64-linux-gnu/
ls -l libv4l2.so.0.0.999999
sudo rm libv4l2.so.0.0.999999
sudo ln -s libv4l2.so.0.0.0.0  libv4l2.so.0.0.999999