Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to install mmdeploy/tensorrt on H100 : Unsupported SM: 0x900 [Bug] #2757

Open
3 tasks done
timsainb opened this issue May 5, 2024 · 0 comments
Open
3 tasks done

Unable to install mmdeploy/tensorrt on H100 : Unsupported SM: 0x900 [Bug] #2757

timsainb opened this issue May 5, 2024 · 0 comments

Comments

@timsainb
Copy link

timsainb commented May 5, 2024 •
edited

Checklist

  • I have searched related issues but cannot get the expected help.
  • 2. I have read the FAQ documentation but cannot get the expected help.
  • 3. The bug has not been fixed in the latest version.

Describe the bug

I am using a new slurm cluster at my university and am unable to get mmdeploy working on the H100 nodes in this cluster. This problem appears to be specific to tensorrt.

I initially tried following the instructions at https://mmdeploy.readthedocs.io/en/latest/get_started.html precisely, then tried a number of permutations:

  • using cuda 11.8 vs 12.1
  • torch 2 vs 1
  • compiling mm packages rather than pip/mim installing
  • using different version of tensorrt, cudnn, onnx, etc

I have installed mmdeploy and been able to install and use mmdeploy on other clusters/gpus on my custom networks in several other environments so I am not sure what's going on here.

Reproduction

I initially followed the instructions at https://mmdeploy.readthedocs.io/en/latest/get_started.html

Note that I am on a slurm cluster at my university.

Here are the specific commands I ran after getting a node.

module load cuda12.1/toolkit/12.1.1
module load  gcc12/12.2.0
conda create --name rtm python=3.8 -y
conda activate rtm
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0rc2"
# 1. install MMDeploy model converter
pip install mmdeploy==1.3.1
# 2. install MMDeploy sdk inference
# you can install one to install according whether you need gpu inference
# 2.1 support onnxruntime
pip install mmdeploy-runtime==1.3.1
# 2.2 support onnxruntime-gpu, tensorrt
pip install mmdeploy-runtime-gpu==1.3.1
# 3. install inference engine
# 3.1 install TensorRT
# !!! If you want to convert a tensorrt model or inference with tensorrt,
# download TensorRT-8.2.3.0 CUDA 11.x tar package from NVIDIA, and extract it to the current directory
pip install TensorRT-8.2.3.0/python/tensorrt-8.2.3.0-cp38-none-linux_x86_64.whl
pip install pycuda
export TENSORRT_DIR=$(pwd)/TensorRT-8.2.3.0
export LD_LIBRARY_PATH=${TENSORRT_DIR}/lib:$LD_LIBRARY_PATH
# !!! Moreover, download cuDNN 8.2.1 CUDA 11.x tar package from NVIDIA, and extract it to the current directory
export CUDNN_DIR=$(pwd)/cuda
export LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
# 3.2 install ONNX Runtime
# you can install one to install according whether you need gpu inference
# 3.2.1 onnxruntime
wget[ https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz](https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz)
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-1.8.1
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
# 3.2.2 onnxruntime-gpu
pip install onnxruntime-gpu==1.8.1
wget[ https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz](https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz)
tar -zxvf onnxruntime-linux-x64-gpu-1.8.1.tgz
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-gpu-1.8.1
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH

cd mmdetection
mim install -v -e .

If I then try to run

python mmdeploy/tools/deploy.py \
    mmdeploy/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    mmdetection/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
    checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    mmdetection/demo/demo.jpg \
    --work-dir mmdeploy_model/faster-rcnn \
    --device cuda \
    --dump-info

I get the error :

File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/site-packages/onnxruntime/capi/_ld_preload.py", line 12, in <module>
    _libcudart = CDLL("libcudart.so.11.0", mode=RTLD_GLOBAL)
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory

I can switch to cuda 11, and try again:

module load cuda11.8/toolkit/11.8.0

This time it runs up until:

[05/05/2024-16:16:18] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +7, GPU +258, now: CPU 761, GPU 1003 (MiB)
[05/05/2024-16:16:18] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +6, GPU +264, now: CPU 767, GPU 1267 (MiB)
[05/05/2024-16:16:18] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[05/05/2024-16:16:20] [TRT] [E] 1: [caskUtils.cpp::trtSmToCask::147] Error Code 1: Internal Error (Unsupported SM: 0x900)
[05/05/2024-16:16:20] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
Process Process-3:
Traceback (most recent call last):
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/site-packages/mmdeploy/apis/utils/utils.py", line 98, in to_backend
    return backend_mgr.to_backend(
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/backend_manager.py", line 127, in to_backend
    onnx2tensorrt(
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/onnx2tensorrt.py", line 79, in onnx2tensorrt
    from_onnx(
  File "/home/tis697/miniconda3/envs/rtm/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/utils.py", line 248, in from_onnx
    assert engine is not None, 'Failed to create TensorRT engine'
AssertionError: Failed to create TensorRT engine
05/05 16:16:21 - mmengine - ERROR - /home/tis697/miniconda3/envs/rtm/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - `mmdeploy.apis.utils.utils.to_backend` with Call id: 1 failed. exit.

If I use a different environment where I used cuda 11.8 instead of cuda 12 I still get this error.

[05/05/2024-17:04:32] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.5 but loaded cuBLAS/cuBLAS LT 111.1.3
[05/05/2024-17:04:32] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +258, now: CPU 748, GPU 1003 (MiB)
[05/05/2024-17:04:32] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +2, GPU +264, now: CPU 750, GPU 1267 (MiB)
[05/05/2024-17:04:32] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[05/05/2024-17:04:34] [TRT] [E] 1: [caskUtils.cpp::trtSmToCask::147] Error Code 1: Internal Error (Unsupported SM: 0x900)
[05/05/2024-17:04:34] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
Process Process-3:
Traceback (most recent call last):
  File "/home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/utils/utils.py", line 98, in to_backend
    return backend_mgr.to_backend(
  File "/home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/backend_manager.py", line 127, in to_backend
    onnx2tensorrt(
  File "/home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/onnx2tensorrt.py", line 79, in onnx2tensorrt
    from_onnx(
  File "/home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/utils.py", line 248, in from_onnx
    assert engine is not None, 'Failed to create TensorRT engine'
AssertionError: Failed to create TensorRT engine
05/05 17:04:35 - mmengine - ERROR - /home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - `mmdeploy.apis.utils.utils.to_backend` with Call id: 1 failed. exit.

If I instead install tensorrt via pypi:

pip install nvidia-pyindex
pip install tensorrt==8.6.1.post1 --extra-index-url https://pypi.nvidia.com

I get a little farther, being able to generate the output, but fail on visualize tensorrt model failed

[05/05/2024-17:17:47] [TRT] [I] Total Activation Memory: 415622144
[05/05/2024-17:17:47] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +64, now: CPU 3456, GPU 1851 (MiB)
[05/05/2024-17:17:47] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +264, now: CPU 3456, GPU 2115 (MiB)
[05/05/2024-17:17:47] [TRT] [W] TensorRT was linked against cuDNN 8.9.0 but loaded cuDNN 8.7.0
[05/05/2024-17:17:47] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +0, GPU +160, now: CPU 0,GPU 160 (MiB)
05/05 17:17:49 - mmengine - INFO - Finish pipeline mmdeploy.apis.utils.utils.to_backend
05/05 17:17:50 - mmengine - INFO - visualize tensorrt model start.
05/05 17:17:54 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
05/05 17:17:54 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
05/05 17:17:54 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "backend_detectors" registry tree. As a workaround, the current "backend_detectors" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
05/05 17:17:54 - mmengine - INFO - Successfully loaded tensorrt plugins from /home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
05/05 17:17:54 - mmengine - INFO - Successfully loaded tensorrt plugins from /home/tis697/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[05/05/2024-17:17:55] [TRT] [W] TensorRT was linked against cuDNN 8.9.0 but loaded cuDNN 8.7.0
[05/05/2024-17:17:55] [TRT] [W] TensorRT was linked against cuDNN 8.9.0 but loaded cuDNN 8.7.0
#assertion/__w/mmdeploy/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/batched_nms/trt_batched_nms.cpp,103
05/05 17:17:56 - mmengine - ERROR - mmdeploy/tools/deploy.py - create_process - 82 - visualize tensorrt model failed.

I can then try to use the models I generated:

$  python
Python 3.8.19 (default, Mar 20 2024, 19:58:24)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> # create object detector
>>> from mmdeploy_runtime import Detector, PoseDetector

loading libmmdeploy_trt_net.so ...
loading libmmdeploy_ort_net.so ...
>>> detector = Detector(model_path="/home/tis697/code/mmlab/mmdeploy_model/faster-rcnn",device_name="cuda")
[2024-05-05 17:21:05.465] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "/home/tis697/code/mmlab/mmdeploy_model/faster-rcnn"
[2024-05-05 17:24:31.661] [mmdeploy] [error] [trt_net.cpp:28] TRTNet: 1: [stdArchiveReader.cpp::StdArchiveReader::30] Error Code 1: Serialization (Serialization assertion magicTagRead == magicTag failed.Magic tag does not match)
[2024-05-05 17:24:31.661] [mmdeploy] [error] [trt_net.cpp:28] TRTNet: 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)
[2024-05-05 17:24:31.661] [mmdeploy] [error] [trt_net.cpp:75] failed to deserialize TRT CUDA engine
[2024-05-05 17:24:31.662] [mmdeploy] [error] [net_module.cpp:54] Failed to create Net backend: tensorrt, config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output"
  ],
  "input_map": {
    "img": "input"
  },
  "is_batched": true,
  "module": "Net",
  "name": "fasterrcnn",
  "output": [
    "infer_output"
  ],
  "output_map": {},
  "type": "Task"
}
[2024-05-05 17:24:31.662] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output"
  ],
  "input_map": {
    "img": "input"
  },
  "is_batched": true,
  "module": "Net",
  "name": "fasterrcnn",
  "output": [
    "infer_output"
  ],
  "output_map": {},
  "type": "Task"
}
>>> import numpy as np
>>> detector(np.zeros((100,100,3)))
SegmentationFault

Environment

$ nvidia-smi
Sun May  5 16:14:17 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          Off |   00000000:52:00.0 Off |                    0 |
| N/A   45C    P0             80W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |


$ python tools/check_env.py
05/05 16:30:05 - mmengine - INFO -

05/05 16:30:05 - mmengine - INFO - **********Environmental information**********
05/05 16:30:07 - mmengine - INFO - sys.platform: linux
05/05 16:30:07 - mmengine - INFO - Python: 3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0]
05/05 16:30:07 - mmengine - INFO - CUDA available: True
05/05 16:30:07 - mmengine - INFO - MUSA available: False
05/05 16:30:07 - mmengine - INFO - numpy_random_seed: 2147483648
05/05 16:30:07 - mmengine - INFO - GPU 0: NVIDIA H100 80GB HBM3
05/05 16:30:07 - mmengine - INFO - CUDA_HOME: /cm/shared/apps/cuda11.8/toolkit/11.8.0
05/05 16:30:07 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.8, V11.8.89
05/05 16:30:07 - mmengine - INFO - GCC: gcc (GCC) 13.1.0
05/05 16:30:07 - mmengine - INFO - PyTorch: 2.1.0
05/05 16:30:07 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.9.2
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

05/05 16:30:07 - mmengine - INFO - TorchVision: 0.16.0
05/05 16:30:07 - mmengine - INFO - OpenCV: 4.9.0
05/05 16:30:07 - mmengine - INFO - MMEngine: 0.10.4
05/05 16:30:07 - mmengine - INFO - MMCV: 2.1.0
05/05 16:30:07 - mmengine - INFO - MMCV Compiler: GCC 9.3
05/05 16:30:07 - mmengine - INFO - MMCV CUDA Compiler: 12.1
05/05 16:30:07 - mmengine - INFO - MMDeploy: 1.3.1+unknown
05/05 16:30:07 - mmengine - INFO -

05/05 16:30:07 - mmengine - INFO - **********Backend information**********
05/05 16:30:08 - mmengine - INFO - tensorrt:    8.2.3.0
05/05 16:30:08 - mmengine - INFO - tensorrt custom ops: Available
05/05 16:30:08 - mmengine - INFO - ONNXRuntime: None
05/05 16:30:08 - mmengine - INFO - ONNXRuntime-gpu:     1.8.1
05/05 16:30:08 - mmengine - INFO - ONNXRuntime custom ops:      Available
05/05 16:30:08 - mmengine - INFO - pplnn:       None
05/05 16:30:08 - mmengine - INFO - ncnn:        None
05/05 16:30:08 - mmengine - INFO - snpe:        None
05/05 16:30:08 - mmengine - INFO - openvino:    None
05/05 16:30:08 - mmengine - INFO - torchscript: 2.1.0
05/05 16:30:08 - mmengine - INFO - torchscript custom ops:      NotAvailable
05/05 16:30:08 - mmengine - INFO - rknn-toolkit:        None
05/05 16:30:08 - mmengine - INFO - rknn-toolkit2:       None
05/05 16:30:08 - mmengine - INFO - ascend:      None
05/05 16:30:08 - mmengine - INFO - coreml:      None
05/05 16:30:08 - mmengine - INFO - tvm: None
05/05 16:30:08 - mmengine - INFO - vacc:        None
05/05 16:30:08 - mmengine - INFO -

05/05 16:30:08 - mmengine - INFO - **********Codebase information**********
05/05 16:30:08 - mmengine - INFO - mmdet:       3.2.0
05/05 16:30:08 - mmengine - INFO - mmseg:       None
05/05 16:30:08 - mmengine - INFO - mmpretrain:  None
05/05 16:30:08 - mmengine - INFO - mmocr:       None
05/05 16:30:08 - mmengine - INFO - mmagic:      None
05/05 16:30:08 - mmengine - INFO - mmdet3d:     None
05/05 16:30:08 - mmengine - INFO - mmpose:      None
05/05 16:30:08 - mmengine - INFO - mmrotate:    None
05/05 16:30:08 - mmengine - INFO - mmaction:    None
05/05 16:30:08 - mmengine - INFO - mmrazor:     None
05/05 16:30:08 - mmengine - INFO - mmyolo:      None

Error traceback

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@timsainb