Real-time pose estimation accelerated with NVIDIA TensorRT

Last update: Jan 06, 2023

Overview

trt_pose

Want to detect hand poses? Check out the new trt_pose_hand project for real-time hand pose and gesture recognition!

trt_pose is aimed at enabling real-time pose estimation on NVIDIA Jetson. You may find it useful for other NVIDIA platforms as well. Currently the project includes

Pre-trained models for human pose estimation capable of running in real time on Jetson Nano. This makes it easy to detect features like left_eye, left_elbow, right_ankle, etc.
Training scripts to train on any keypoint task data in MSCOCO format. This means you can experiment with training trt_pose for keypoint detection tasks other than human pose.

To get started, follow the instructions below. If you run into any issues please let us know.

Getting Started

To get started with trt_pose, follow these steps.

Step 1 - Install Dependencies

Install PyTorch and Torchvision. To do this on NVIDIA Jetson, we recommend following this guide

Install torch2trt

git clone https://github.com/NVIDIA-AI-IOT/torch2trt
cd torch2trt
sudo python3 setup.py install --plugins

Install other miscellaneous packages

sudo pip3 install tqdm cython pycocotools
sudo apt-get install python3-matplotlib

Step 2 - Install trt_pose

git clone https://github.com/NVIDIA-AI-IOT/trt_pose
cd trt_pose
sudo python3 setup.py install

Step 3 - Run the example notebook

We provide a couple of human pose estimation models pre-trained on the MSCOCO dataset. The throughput in FPS is shown for each platform

Model	Jetson Nano	Jetson Xavier	Weights
resnet18_baseline_att_224x224_A	22	251	download (81MB)
densenet121_baseline_att_256x256_B	12	101	download (84MB)

To run the live Jupyter Notebook demo on real-time camera input, follow these steps

Download the model weights using the link in the above table.
Place the downloaded weights in the tasks/human_pose directory
Open and follow the live_demo.ipynb notebook

You may need to modify the notebook, depending on which model you use

References

The trt_pose model architectures listed above are inspired by the following works, but are not a direct replica. Please review the open-source code and configuration files in this repository for architecture details. If you have any questions feel free to reach out.

Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
Xiao, Bin, Haiping Wu, and Yichen Wei. "Simple baselines for human pose estimation and tracking." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

Comments

Increase FPS

I am using trt_pose on a system with i5 cpu and GTX 1660 Super GPU. I am currently only getting 9 fps for an 1080 x 1920 video. Any ideas for how to improve FPS?

opened by mcdonasd 5
replacing jetcam with opencv works but only shows image without any drawing on it.

Hello everyone,

this issue is redundant, I re-post it due the misleading title in the original one #68

this is the live_demo with opencv, just change the the file format to ipynb (github doesn't allow ipynb files attached): live_demo.txt

I'm feeding the image to the execute method, but the count of people is always 0(which is wrong).

opened by Hasankanso 5
trt_pose not utilizing the GPU on jetson Tx2

Hello All,

We are trying to develop an application which uses trt_pose, the model is too slow and the max we could extract was 15fps on Jetson Tx2, after nvpmodel and jetson_clocks. Also, I observed that through tegrastats, the code is not utilizing all the cores, and I believe the GPU cores have zero usage during runtime, even though it is enabled inside the code. Can you please help me in getting a boost in the performance? Thank you.

opened by AmoghBharadwaj 5

Import Error

Hello, Has anyone happen to have gotten this error when importing trt_pose.coco before?

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
 in 
      1 import json
----> 2 import trt_pose.coco
      3 
      4 with open('human_pose.json', 'r') as f:
      5     human_pose = json.load(f)

/usr/local/lib/python3.6/dist-packages/trt_pose-0.0.1-py3.6-linux-aarch64.egg/trt_pose/coco.py in 
      7 import tqdm
      8 import trt_pose
----> 9 import trt_pose.plugins
     10 import glob
     11 import torchvision.transforms.functional as FT

ImportError: /usr/local/lib/python3.6/dist-packages/trt_pose-0.0.1-py3.6-linux-aarch64.egg/trt_pose/plugins.cpython-36m-aarch64-linux-gnu.so: undefined symbol: _ZTIN3c1021AutogradMetaInterfaceE

opened by Daniel-10 4

How can I train trt_pose on multi-GPU?

@jaybdub Hi, jaybdub, Thanks for your great work! When I was trainning the net, I find the trainning process on only one GPU(I have two), how can I train trt_pose on multi-GPU? And restore from a pretrained checkpoint?

opened by kinglintianxia 4
Low FPS than report in Jetson nano

Hi, I highly appreciate your repo. But I test your model in Jetson nano Tegra X1 it just got about 8FPS lower than 22 FPS in your report. I just follow your guide except I did not use USB camera, I used OpenCV to read image instead of. I want to know whether you use other kind of Jetson nano, which version of Pytorch or I made a mistake. Many tks.

opened by tucachmo2202 3
RuntimeError: NVRTC error: while running cell-5

While running cell-5 of live_demo.ipynb notebook import torch2trt model_trt = torch2trt.torch2trt(model, [data], fp16_mode=False, max_workspace_size=1<<25)
getting error below: RuntimeError Traceback (most recent call last) in 1 import torch2trt 2 ----> 3 model_trt = torch2trt.torch2trt(model, [data], fp16_mode=False, max_workspace_size=1<<25)

~/tlt-experiments/skeleton-env/lib/python3.6/site-packages/torch2trt-0.1.0-py3.6-linux-x86_64.egg/torch2trt/torch2trt.py in torch2trt(module, inputs, input_names, output_names, log_level, max_batch_size, fp16_mode, max_workspace_size, strict_type_constraints, keep_network, int8_mode, int8_calib_dataset, int8_calib_algorithm, int8_calib_batch_size, use_onnx) 553 ) 554 --> 555 engine = builder.build_cuda_engine(network) 556 557 module_trt = TRTModule(engine, input_names, output_names)

RuntimeError: NVRTC error:

opened by arvindchandel 3
Getting pose metadata

Hi @jaybdub!

Simply awesome the way we can detect poses and plot it on vídeo!! Great job!!

I just couldn't find out how to get pose's features metadata programatically in python (eg.: left_eye, left_elbow, right_ankle, etc.)

Could you please clarify it?

Thanks!

opened by rshigemura 3
How to continue training from checkpoint?

Dear Nvidia team, I would like to do some tranfer learning with my own data based on the model already trained on COCO dataset. However, I only found that I can change the checkpoint interval in the training config file. But had no clue about how to continue training from a checkpoint file. Could you please give me some idea about this if possible? Many thanks!

opened by Tony-Yan2018 2
import-im6.q16: not authorized `re'

errors occur when convert .pth to onnx . I just follow the deep_stream_trt_pose instrution use : ./export_for_isaac.py --input_checkpoint resnet18_baseline_att_224x224_A_epoch_249.pth erros: : File name too long import-im6.q16: not authorized re' @ error/constitute.c/WriteImage/1037. import-im6.q16: not authorizedtorch' @ error/constitute.c/WriteImage/1037. ./export_for_isaac.py: line 60: syntax error near unexpected token (' ./export_for_isaac.py: line 60:class InputReNormalization(torch.nn.Module):'

opened by bobby20180331 2
Cuda

Hi All I successfully installed trt_pose on jetson nano board, very interesting, thanks! i would like to install on a pc with 2070rtx using nvcr.io/nvidia/tensorrt:20.12-py3 based custom docker but, during model conversion (early), it gives me this error

[TensorRT] ERROR: CUDA initialization failure with error 222. Please check your CUDA installation: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html Traceback (most recent call last): File "live_demo.py", line 40, in model_trt = torch2trt.torch2trt(model, [data], fp16_mode=True, max_workspace_size=1<<25) File "/usr/local/lib/python3.8/dist-packages/torch2trt-0.1.0-py3.8.egg/torch2trt/torch2trt.py", line 498, in torch2trt TypeError: pybind11::init(): factory function returned nullptr

ubuntu: 20.04 Driver Version: 450.80.02 cuda: 11.1 torch: 1.7.1 torch2trt: 0.1.0 torchvision: 0.8.2

Thanks in advance Gianluca

opened by dexmac221 2
Runtime Error: Error compiling objects for extension

The stack trace is as follows:

raceback (most recent call last): File "setup.py", line 52, in setup( File "/usr/local/lib/python3.8/dist-packages/setuptools/init.py", line 87, in setup return distutils.core.setup(**attrs) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/usr/local/lib/python3.8/dist-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py", line 74, in run self.do_egg_install() File "/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py", line 123, in do_egg_install self.run_command('bdist_egg') File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/usr/local/lib/python3.8/dist-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/local/lib/python3.8/dist-packages/setuptools/command/bdist_egg.py", line 165, in run cmd = self.call_command('install_lib', warn_dir=0) File "/usr/local/lib/python3.8/dist-packages/setuptools/command/bdist_egg.py", line 151, in call_command self.run_command(cmdname) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/usr/local/lib/python3.8/dist-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/local/lib/python3.8/dist-packages/setuptools/command/install_lib.py", line 11, in run self.build() File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/install_lib.py", line 112, in build self.run_command('build_ext') File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/usr/local/lib/python3.8/dist-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/local/lib/python3.8/dist-packages/setuptools/command/build_ext.py", line 84, in run _build_ext.run(self) File "/usr/local/lib/python3.8/dist-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build_ext.py", line 346, in run self.build_extensions() File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 741, in build_extensions build_ext.build_extensions(self) File "/usr/local/lib/python3.8/dist-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions self._build_extensions_serial() File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial self.build_extension(ext) File "/usr/local/lib/python3.8/dist-packages/setuptools/command/build_ext.py", line 246, in build_extension _build_ext.build_extension(self, ext) File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build_ext.py", line 549, in build_extension objects = self.compiler.compile( File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 562, in unix_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 1419, in _write_ninja_file_and_compile_objects _run_ninja_build( File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 1756, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension

opened by mjehanzaibxvr 1
Code fails to run in both jupyter and console

Sorry for the less than descriptive title, but I wasn't sure how else to title it.

I've got a 4gb Jetson Nano (SeeedStudio Jetson Recomputer J2010 carrier board) with a 128gb SSD as the root storage device. It's running JetPack 4.6 (output of 'apt-cache show nvidia-jetpack' below)

`$ sudo apt-cache show nvidia-jetpack Package: nvidia-jetpack Version: 4.6-b199 Architecture: arm64 Maintainer: NVIDIA Corporation Installed-Size: 194 Depends: nvidia-cuda (= 4.6-b199), nvidia-opencv (= 4.6-b199), nvidia-cudnn8 (= 4.6-b199), nvidia-tensorrt (= 4.6-b199), nvidia-visionworks (= 4.6-b199), nvidia-container (= 4.6-b199), nvidia-vpi (= 4.6-b199), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0) Homepage: http://developer.nvidia.com/jetson Priority: standard Section: metapackages Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b199_arm64.deb Size: 29368 SHA256: 69df11e22e2c8406fe281fe6fc27c7d40a13ed668e508a592a6785d40ea71669 SHA1: 5c678b8762acc54f85b4334f92d9bb084858907a MD5sum: 1b96cd72f2a434e887f98912061d8cfb Description: NVIDIA Jetpack Meta Package Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack Version: 4.6-b197 Architecture: arm64 Maintainer: NVIDIA Corporation Installed-Size: 194 Depends: nvidia-cuda (= 4.6-b197), nvidia-opencv (= 4.6-b197), nvidia-cudnn8 (= 4.6-b197), nvidia-tensorrt (= 4.6-b197), nvidia-visionworks (= 4.6-b197), nvidia-container (= 4.6-b197), nvidia-vpi (= 4.6-b197), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0) Homepage: http://developer.nvidia.com/jetson Priority: standard Section: metapackages Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b197_arm64.deb Size: 29356 SHA256: 104cd0c1efefe5865753ec9b0b148a534ffdcc9bae525637c7532b309ed44aa0 SHA1: 8cca8b9ebb21feafbbd20c2984bd9b329a202624 MD5sum: 463d4303429f163b97207827965e8fe0 Description: NVIDIA Jetpack Meta Package Description-md5: ad1462289bdbc54909ae109d1d32c0a8 ` I've set up a python 3.6 virtualenv and followed the installation instructions for all required packages. There were no errors during any of the installations and I've verified all of the packages import properly from the python command line. When I run the jupyter notebook 'live_demo.ipynb' I am able to run all of the steps up until the following step:

`import torch2trt

model_trt = torch2trt.torch2trt(model, [data], fp16_mode=True, max_workspace_size=1<<25)`

When I attempt to run that step the systems thinks about it for a bit and then a dialog pops up that says the python kernel has crashed and will be automatically restarted. I cannot get past this step.

To help debug/diagnose I took all of the code from the notebook and incrementally added it to a python file to see if I could reproduce the issue. The code I have so far is:

`import cv2 import json import trt_pose.coco import trt_pose.models import torch import torch2trt from torch2trt import TRTModule import time import torchvision.transforms as transforms import PIL.Image from trt_pose.draw_objects import DrawObjects from trt_pose.parse_objects import ParseObjects from jetcam.usb_camera import USBCamera from jetcam.csi_camera import CSICamera from jetcam.utils import bgr8_to_jpeg import ipywidgets from IPython.display import display

with open('human_pose.json', 'r') as f: human_pose = json.load(f)

topology = trt_pose.coco.coco_category_to_topology(human_pose)

num_parts = len(human_pose['keypoints']) num_links = len(human_pose['skeleton'])

model = trt_pose.models.resnet18_baseline_att(num_parts, 2 * num_links).cuda().eval() MODEL_WEIGHTS = 'resnet18_baseline_att_224x224_A_epoch_249.pth' model.load_state_dict(torch.load(MODEL_WEIGHTS))

WIDTH = 224 HEIGHT = 224

data = torch.zeros((1, 3, HEIGHT, WIDTH)).cuda() print("Calling torch2trt.torch2trt\n") model_trt = torch2trt.torch2trt(model, [data], fp16_mode=True, max_workspace_size=1<<25) print("Done\n") OPTIMIZED_MODEL = 'resnet18_baseline_att_224x224_A_epoch_249_trt.pth' print("Calling torch.save\n") torch.save(model_trt.state_dict(), OPTIMIZED_MODEL) print("Done\n")`

When I run this code I see "Calling torch2trt.torch2trt" in the console and then, after a pause, I see the following error repeated many times on the console:

[TensorRT] ERROR: 3: [builderConfig.cpp::canRunOnDLA::341] Error Code 3: Internal Error (Parameter check failed at: optimizer/api/builderConfig.cpp::canRunOnDLA::341, condition: dlaEngineCount > 0 )

After that the system will seem to hang. The desktop will display a low memory warning (If I run 'watch -n1 free -h' in another console window I can see free memory drop from 3+GB to as little as 96MB). After some time the process just reports "Killed" and exits back to the command line.

I am at a loss. Can you please provide any helpful information you have that might help me correct this issue and continue?

Thanks Marc

opened by marcjasner 0
Resume training from checkpoint and Start training from pretrained model
Hi @jaybdub ,

Does this project have the feature to resume training from a checkpoint?

Are we able to start training from a pretrained model on a new dataset?

Thank you!
opened by JJLimmm 0
parmeter check failled.

my running code is failed after reinstall the dependency. my jet pack details :- NVIDIA Jetson UNKNOWN L4T 32.5.0 [ JetPack 4.5 ] Ubuntu 18.04.5 LTS Kernel Version: 4.9.201+ CUDA 10.2.300 CUDA Architecture: NONE OpenCV version: 4.1.1 OpenCV Cuda: NO CUDNN: 8.0.0.180 TensorRT: 7.1.3.0 Vision Works: 1.6.0.501 VPI: ii libnvvpi1 1.0.12 arm64 NVIDIA Vision Programming Interface library Vulcan: 1.2.70

opened by abhijithyes 0
Evaluation metrics for model

Hi, i would like to ask if there is any available evaluation metrics for this model? There is a eval.ipynb notebook here but it seems incomplete and using normal object detection metrics for evaluating the model.

Is there any scripts available to use for evaluating the keypoint detection for metrics like PCK, PDJ, PCP as shown in this repo ?

Thanks!

opened by JJLimmm 0