PROBLEM ABOUT THE DEPLOYMENT OF THE MODEL TO THE HARDWARE.

[RonahJay-Emperio]

Search before asking

• I have searched the HUB issues and found no similar bug report.

HUB Component

Models

Bug

Hi! I tried to deploy the model to the hardware and it seems like the hardware cannot detect the classes based on the model that I created. I tried to test the model in the Ultralytics app and it works perfectly. My main concern is, how can we fix this issue and what are the possible cause of this error? I hope to get a fast response from you about this matter. Thank You!

Environment

1. I use the iOS application for testing the models.
2. I used tensorFlow lite, ONNX, OpenVINO for deployment. But, none of these conversions worked.

Minimal Reproducible Example

No response

Additional

No response

[github-actions]

👋 Hello @RonahJay-Emperio, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

[kalenmike]

@RonahJay-Emperio Thanks for raising the question. We are not able to help as we cannot see your code. We are currently working on getting the Ultralytics App code ready for open source, in the mean time we have a similair app already open sourced for iOS. My advice would be to take a look at the repo and see if there are any differences to how we load and run the model:

https://github.com/ultralytics/yolo-ios-app

[RonahJay-Emperio]

Hi, @kalenmike! Thank You for your response. I used the Luxonis OAK FFC 4P device and it seems that the detection is not accurate because there's a lot of bounding boxes that are being detected during deployment. How can I detect the classes one at a time? I would really appreciate it if you can help me with this issue. Thank You.

Here's the process I made before deployment:

1. Convert ONNX model into blob file.
2. Deploy the converted model into the device which is the OAK FFC 4P

Here's the sample code that I used:

from pathlib import Path
import cv2
import depthai as dai
import numpy as np
import time
import argparse

nnPathDefault = str((Path(file).parent / Path('../models/worker_activity_openvino_2022.1_6shave.blob')).resolve().absolute())
parser = argparse.ArgumentParser()
parser.add_argument('nnPath', nargs='?', help="Path to mobilenet detection network blob", default=nnPathDefault)
parser.add_argument('-s', '--sync', action="store_true", help="Sync RGB output with NN output", default=False)
args = parser.parse_args()

if not Path(nnPathDefault).exists():
import sys
raise FileNotFoundError(f'Required file/s not found, please run "{sys.executable} install_requirements.py"')

MobilenetSSD label texts

labelMap = ["Walking", "At-Desk-Working", "Fall", "Running", "Sleeping", "Standing-NotWorking", "Standing-Working", "At-Desk-NotWorking"]

Create pipeline

pipeline = dai.Pipeline()

Define sources and outputs

camRgb = pipeline.create(dai.node.ColorCamera)
nn = pipeline.create(dai.node.NeuralNetwork)
det = pipeline.create(dai.node.DetectionParser)
xoutRgb = pipeline.create(dai.node.XLinkOut)
nnOut = pipeline.create(dai.node.XLinkOut)

xoutRgb.setStreamName("rgb")
nnOut.setStreamName("nn")

Properties

camRgb.setPreviewSize(320, 320)
camRgb.setInterleaved(False)
camRgb.setFps(40)

Define a neural network that will make predictions based on the source frames

nn.setNumInferenceThreads(2)
nn.input.setBlocking(False)

blob = dai.OpenVINO.Blob(args.nnPath)
nn.setBlob(blob)
det.setBlob(blob)
det.setNNFamily(dai.DetectionNetworkType.MOBILENET)
det.setConfidenceThreshold(1.0)

Linking

if args.sync:
nn.passthrough.link(xoutRgb.input)
else:
camRgb.preview.link(xoutRgb.input)

camRgb.preview.link(nn.input)
nn.out.link(det.input)
det.out.link(nnOut.input)

Connect to device and start pipeline

with dai.Device(pipeline) as device:
# Output queues will be used to get the rgb frames and nn data from the outputs defined above
qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
qDet = device.getOutputQueue(name="nn", maxSize=4, blocking=False)

frame = None
detections = []
startTime = time.monotonic()
counter = 0
color2 = (255, 255, 255)


# nn data (bounding box locations) are in <0..1> range - they need to be normalized with frame width/height
def frameNorm(frame, bbox):
    normVals = np.full(len(bbox), frame.shape[0])
    normVals[::2] = frame.shape[1]
    return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)


def displayFrame(name, frame):
    color = (255, 0, 0)
    for detection in detections:
        if detection.label in range(7):
            bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
            cv2.putText(frame, labelMap[detection.label], (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX,0.5, color)
            # cv2.putText(frame, str(detection.label), (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, color)
            cv2.putText(frame, f"{int(detection.confidence * 100)}%", (bbox[0] + 10, bbox[1] + 40),cv2.FONT_HERSHEY_TRIPLEX, 0.5, color)
            cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color, 2)
    # Show the frame
    cv2.imshow(name, frame)


while True:
    if args.sync:
        # Use blocking get() call to catch frame and inference result synced
        inRgb = qRgb.get()
        inDet = qDet.get()
    else:
        # Instead of get (blocking), we use tryGet (non-blocking) which will return the available data or None otherwise
        inRgb = qRgb.tryGet()
        inDet = qDet.tryGet()

    if inRgb is not None:
        frame = inRgb.getCvFrame()
        cv2.putText(frame, "NN fps: {:.2f}".format(counter / (time.monotonic() - startTime)),
                    (2, frame.shape[0] - 4), cv2.FONT_HERSHEY_TRIPLEX, 0.4, color2)

    if inDet is not None:
        detections = inDet.detections
        counter += 1

    # If the frame is available, draw bounding boxes on it and show the frame
    if frame is not None:
        displayFrame("rgb", frame)

    if cv2.waitKey(1) == ord('q'):
        break

[pderrenger]

Hi @RonahJay-Emperio! It sounds like the issue you're experiencing might be related to overlapping detections and the confidence threshold of your model. 🤔

Given the details you've provided, here are a couple of relatively straightforward steps you can experiment with to address the problem of multiple bounding boxes and ensure your model detects classes more distinctly:

1. Adjust the Confidence Threshold: Your current confidence threshold is set to 1.0 in the setConfidenceThreshold(1.0) method. This might be too high and could be the reason why you're seeing multiple bounding boxes for the same object. Typically, a threshold of 0.5 or even lower is used. Try lowering this threshold to see if you get fewer, more accurate detections.

2. Non-Maximum Suppression (NMS): Ensure that Non-Maximum Suppression (NMS) is correctly applied. NMS helps in reducing the number of overlapping bounding boxes. Verify if your deployment environment (in your case, the OAK FFC 4P device setup) is applying NMS effectively after detection. Sometimes, manual adjustment or ensuring that the deployment library supports NMS could be necessary.

Since it looks like you're using the DepthAI library with OAK, double-check the DepthAI documentation or forums for specific advice on optimizing detections and applying NMS correctly for your device.

Keep in mind adjusting the confidence threshold is often the first step and can significantly reduce unwanted detections. If the issue persists, it might be beneficial to explore other preprocessing steps or even retrain your model with varied data to improve its robustness against false positives.

I hope this helps! If you're still facing issues, providing details about the detection results post-adjustment could be useful for further diagnosis. 🛠️

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐