Improving YOLO NAS inference time

[Idoleshem]

Hi Mikel, amazing repo!
I would appreciate your help with a quick question :)

I tried to test YOLO_NAS_S + STRONGSORT performance and the inference time was much higher than what Deci had reported.
My results were around ±120 ms per image, while yolov8n was around 15ms.

Do you any idea how can I address this latency issue and improve the performance?
I turned off all the parameters which responsible for saving images and the cropped_id, but no measure improvement was observed.

Thanks!

[mikel-brostrom]

It seems like there's a lot of overhead in the model.predict() function... Deci-AI/super-gradients#958

[mikel-brostrom]

I manged to improve the yolo_nas inference x2. #1098. Should be running at ~60ms per image. But the overhead in the computations for .pt inference is still quite big and it is a known issue.

[IdoLeshem1]

Thanks! I appreciate that :)

[goldmyu]

Any news regarding that issue @mikel-brostrom ? it really hinders the performance that Deci are promising with Yolo-NAS

[mikel-brostrom]

This issue is still there. Deci-AI/super-gradients#958 (comment). No answers

[Sumeshbaba]

Hello Mikel! Any update regarding this issue? The speed still seems to be quite low on PyTorch.

[mikel-brostrom]

Nope 😢. It does not seem to be a priority for Deci to fix this

[Sumeshbaba]

This seems quite counterintuitive! When YOLOv7 and v8 can perform at about 15-20 ms, YOLO-NAS taking about 60-80 ms on the same device is not ideal! Hope they work on it soon!