ERROR - Calculate multi batch perspective transformation matrix failed!

[abhay-iy97]

Hello,
I am working on integrating the nvocdr model with Triton on a Jetson orin device on JP5.1.2. Inferences of most images work fine however for few images, I get the following errors on the server side.

stderr

[ERROR] Calculate multi batch perspective transformation matrix failed!
[ERROR] Rectification falied. OCRNet inference will not launch

After a few more occurrences of the errors mentioned above I also observe the following log in the stderr of the TritonServer

E0426 22:29:54.663359 19 python_be.cc:2106] Stub process is unhealthy and it will be restarted.

stdout

execution for matrix 00007 is successful, but U is singular and U(6,6) = 0..

On the client side, this is the error I receive from Triton

Traceback (most recent call last):
  File "xx.py", line 88, in function
    results = triton_client.infer(model_name=model_name, inputs=inputs, outputs=outputs)
  File "/usr/local/lib/python3.8/site-packages/tritonclient/grpc/_client.py", line 1572, in infer
    raise_error_grpc(rpc_error)
  File "/usr/local/lib/python3.8/site-packages/tritonclient/grpc/_utils.py", line 77, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] Failed to process the request(s) for model instance 'nvOCDR', message: MemoryError: std::bad_alloc

At:
  /opt/nvocdr/ocdr/triton/models/nvOCDR/1/model.py(169): execute

Can you please provide some more intuition regarding what may cause this error for an image?

[Tyler-D]

This is mostly due to the wrong text detection in OCDNet. I suggest you dump the bbox predicted by OCDNet at this point: https://github.com/NVIDIA-AI-IOT/NVIDIA-Optical-Character-Detection-and-Recognition-Solution/blob/main/src/OCDNetEngine.cpp#L213

Check and visualize them on the image to see if there is something wrong.