Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element

[SIGMIND]

I am trying to run the GazeNet TAO sample app on Jetson with following configuration:

NVIDIA Jetson AGX Xavier [16GB]
 L4T 32.6.1 [ JetPack 4.6 ]
   Ubuntu 18.04.5 LTS
   Kernel Version: 4.9.253-tegra
 CUDA 10.2.300
   CUDA Architecture: 7.2
 OpenCV version: 4.1.1
   OpenCV Cuda: NO
 CUDNN: 8.2.1.32
 TensorRT: 8.0.1.6
 Vision Works: 1.6.0.501
 VPI: ii libnvvpi1 1.1.12 arm64 NVIDIA Vision Programming Interface library
 Vulcan: 1.2.70

When I run the app with the command as mentioned in ReadME (with default sample config files)
./deepstream-gaze-app 2 /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/configs/facial_tao/sample_faciallandmarks_config.txt file:///home/nvidia/Pictures/face.jpg ./gazenet

It crashes with the following error:

Request sink_0 pad from streammux
Now playing: file:///home/nvidia/Pictures/face.jpg
Library Opened Successfully
Setting custom lib properties # 1
Adding Prop: config-file : ../../../configs/gaze_tao/sample_gazenet_model_config.txt
Inside Custom Lib : Setting Prop Key=config-file Value=../../../configs/gaze_tao/sample_gazenet_model_config.txt
0:00:03.325956146 22185   0x55a517f8f0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 2]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
INFO: [FullDims Engine Info]: layers num: 3
0   INPUT  kFLOAT input_face_images:0 1x80x80         min: 1x1x80x80       opt: 32x1x80x80      Max: 32x1x80x80      
1   OUTPUT kFLOAT softargmax/strided_slice_1:0 80              min: 0               opt: 0               Max: 0               
2   OUTPUT kFLOAT softargmax/strided_slice:0 80x2            min: 0               opt: 0               Max: 0               

0:00:03.326175740 22185   0x55a517f8f0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 2]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
0:00:03.348156671 22185   0x55a517f8f0 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<second-infer-engine1> [UID 2]: Load new model:../../../configs/facial_tao/faciallandmark_sgie_config.txt sucessfully
0:00:03.348498992 22185   0x55a517f8f0 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1161> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
0:00:03.384695424 22185   0x55a517f8f0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x416x736       
1   OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46         
2   OUTPUT kFLOAT output_cov/Sigmoid 1x26x46         

0:00:03.384832039 22185   0x55a517f8f0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
0:00:03.387896538 22185   0x55a517f8f0 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-infer-engine1> [UID 1]: Load new model:../../../configs/facial_tao/config_infer_primary_facenet.txt sucessfully
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: nvjpegdec0
In cb_newpad
###Decodebin pick nvidia decoder plugin.
0:00:03.417211615 22185   0x55a5158d40 WARN          nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
0:00:03.417257217 22185   0x55a5158d40 WARN          nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: surface-gpu-id=604095472,source_nvvidconv-gpu-id=0
0:00:03.417366983 22185   0x55a5158d40 ERROR         nvvideoconvert gstnvvideoconvert.c:3484:gst_nvvideoconvert_transform: buffer transform failed
ERROR from element source_nvvidconv: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvvideoconvert/gstnvvideoconvert.c(3098): gst_nvvideoconvert_transform (): /GstPipeline:pipeline/GstBin:source-bin-00/Gstnvvideoconvert:source_nvvidconv:
surface-gpu-id=604095472,source_nvvidconv-gpu-id=0
Returned, stopping playback
Deserializing engine from: ./gazeinfer_impl/../../../../models/gazenet/gazenet_facegrid.etlt_b8_gpu0_fp16.engineThe logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. TensorRT maintains only a single logger pointer at any given time, so the existing value, which can be retrieved with getLogger(), will be used instead. In order to use a new logger, first destroy all existing builder, runner or refitter objects.

Average fps 0.000233
Totally 0 faces are inferred
Deleting pipeline

[njnumjn]

If you want to change to GPU 1, could you try to set environment CUDA_VISIBLE_DEVICES before run your application:
export CUDA_VISIBLE_DEVICES=1, make sure the gpu-id in configuration is 0, or it will run failed:

(deepstream-test5-app:705150): GLib-GObject-CRITICAL **: 11:29:02.143: g_object_set: assertion 'G_IS_OBJECT (object)' failed
Unable to set device in gst_nvstreammux_change_state
Unable to set device in gst_nvstreammux_change_state
Unable to set device in gst_nvstreammux_change_state
** ERROR: main:1573: Failed to set pipeline to PAUSED
Quitting
Unable to set device in gst_nvstreammux_change_state
App run failed

Reference:

https://forums.developer.nvidia.com/t/use-gpu-id-1-in-deepstream-app/204191