TypeError: pybind11::init(): factory function returned nullptr

[majonathany]

Description

When trying to load an ONNX model from samples using following code, I get TypeError: pybind11::init(): factory function returned nullptr.

The following is a script (newfile3.py) that points to an ONNX file:

import os, onnx, onnx_tensorrt
import onnx_tensorrt.backend as backend

onnx_path = os.path.join("/home/ubuntu/my_project", "yolo.onnx")
onnx_model = onnx.load(onnx_path)
engine = backend.prepare(onnx_model, device='CUDA:0')

The file yolo.onnx was taken from the onnx_sample repository: https://github.com/onnx/models
When trying to build the engine, I get the following:

[09/26/2023-19:53:25] [TRT] [W] Unable to determine GPU memory usage
[09/26/2023-19:53:25] [TRT] [W] Unable to determine GPU memory usage
[09/26/2023-19:53:25] [TRT] [W] CUDA initialization failure with error: 35. Please check your CUDA installation: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
Traceback (most recent call last):
File "/home/ubuntu/my_project/newfile3.py", line 6, in
engine = backend.prepare(onnx_model, device='CUDA:0')
File "/home/ubuntu/venv/lib/python3.8/site-packages/onnx_tensorrt-8.6.1-py3.8.egg/onnx_tensorrt/backend.py", line 238, in prepare
File "/home/ubuntu/venv/lib/python3.8/site-packages/onnx_tensorrt-8.6.1-py3.8.egg/onnx_tensorrt/backend.py", line 42, in init
TypeError: pybind11::init(): factory function returned nullptr

However, I can import tensorrt, onnx and onnx_tensorrt separately fine, and when I run: /usr/src/tensorrt/bin/trtexec, I get a "PASSED" message.

I get this same message when I run the "onnx_backend_test.py" test.

Environment

TensorRT Version: tensorrt.version (in python) produces '8.6.1'
ONNX-TensorRT Version / Branch: main
GPU Type: ec2 G5 A10G, NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6
Nvidia Driver Version: 510.47.03
CUDA Version: 11.6
CUDNN Version: 8.9.5
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.8.12
TensorFlow + TF2ONNX Version (if applicable): not using tensorflow
PyTorch Version (if applicable): 1.13
Baremetal or Container (if container which image + tag): baremetal

Relevant Files

output from nvidia-smi:

Tue Sep 26 20:02:42 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 |
| 0% 33C P0 61W / 300W | 247MiB / 23028MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2300 C python3 245MiB |
+-----------------------------------------------------------------------------+

Steps To Reproduce

Packages installed:
cudnn-local-repo-ubuntu1804-8.9.5.29_1.0-1_amd64.deb
nv-tensorrt-local-repo-ubuntu1804-8.6.1-cuda-11.8_1.0-1_amd64.deb

The following is the ~/onnx-tensorrt/build folder:

~/onnx-tensorrt/build

drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 25 21:56 third_party
drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 25 22:08 lib
-rw-r--r-- 1 root root 3828 Sep 26 19:07 CPackSourceConfig.cmake
-rw-r--r-- 1 root root 3373 Sep 26 19:07 CPackConfig.cmake
-rw-r--r-- 1 root root 3825 Sep 26 19:07 cmake_install.cmake
lrwxrwxrwx 1 root root 24 Sep 26 19:08 libnvonnxparser.so.8 -> libnvonnxparser.so.8.6.1
lrwxrwxrwx 1 root root 20 Sep 26 19:08 libnvonnxparser.so -> libnvonnxparser.so.8
-rwxr-xr-x 1 root root 1345992 Sep 26 19:08 libnvonnxparser.so.8.6.1
-rw-r--r-- 1 root root 1819670 Sep 26 19:08 libnvonnxparser_static.a
-rw-r--r-- 1 root root 22533 Sep 26 19:21 CMakeCache.txt
-rw-r--r-- 1 root root 26285 Sep 26 19:21 Makefile
drwxr-xr-x 9 root root 4096 Sep 26 19:51 CMakeFiles
-rw-r--r-- 1 root root 149 Sep 26 19:51 install_manifest.txt
drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 26 19:52 bdist.linux-x86_64

[wanglong001]

+1

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 522.30       CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  N/A |
| N/A   49C    P8    19W /  N/A |   1480MiB /  6144MiB |     50%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

import onnx
import onnx_tensorrt.backend as backend
import numpy as np

model = onnx.load("model.onnx")
engine = backend.prepare(model, device="CUDA")
input_data = np.random.random(size=(1, 3, 224, 224)).astype(np.float32)
output_data = engine.run(input_data)[0]
print(output_data)
print(output_data.shape)

root@5644f2ca731e:/opt/test# python3 test_onnx.py
[11/22/2023-07:29:18] [TRT] [W] Unable to determine GPU memory usage
[11/22/2023-07:29:18] [TRT] [W] Unable to determine GPU memory usage
[11/22/2023-07:29:18] [TRT] [W] CUDA initialization failure with error: 35. Please check your CUDA installation:  http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
Traceback (most recent call last):
  File "/opt/test/test_onnx.py", line 6, in <module>
    engine = backend.prepare(model, device="CUDA")
  File "/usr/local/lib/python3.10/dist-packages/onnx_tensorrt-8.6.1-py3.10.egg/onnx_tensorrt/backend.py", line 238, in prepare
  File "/usr/local/lib/python3.10/dist-packages/onnx_tensorrt-8.6.1-py3.10.egg/onnx_tensorrt/backend.py", line 42, in __init__
TypeError: pybind11::init(): factory function returned nullptr