modes/export/

[giscus[bot]]

modes/export/

Step-by-step guide on exporting your YOLOv8 models to various format like ONNX, TensorRT, CoreML and more for deployment. Explore now!.

https://docs.ultralytics.com/modes/export/

[alxAgu]

Where can we find working examples of a tf.js exported model?

[pderrenger]

@alxAgu currently, we don't have specific examples of a tf.js exported model directly within the YOLOv8 documentation. However, exporting a model to TensorFlow.js (tf.js) format typically involves converting the model to a format compatible with tf.js and then loading and running it within a web environment. For the export part you can use i.e.

yolo export model=yolov8n.pt format=tensorflowjs

For general guidance on how to use a tf.js model, you can refer to the TensorFlow.js documentation and tutorials, which provide comprehensive instructions on how to import and run models in the browser or Node.js.

If you're looking for examples specific to YOLO models, you might find community-contributed projects or tutorials by searching online for "YOLO TensorFlow.js example" or similar queries. These resources can often provide practical demonstrations of how to integrate YOLO models with tf.js.

[alxAgu]

@pderrenger, are there any plans to provide a working example of a YOLOv8 model exported to work in the browser with TensorFlow.js? There's a big appetite for this and it's honestly the only thing preventing us from recommending Ultralytics HUB to our clients.

[pderrenger]

Hi @alxAgu, thanks for reaching out! 🚀 We currently don't have a specific example for YOLOv8 models exported to work with TensorFlow.js. However, we appreciate the feedback and understand the demand for such examples. Your suggestion is definitely on our radar, and we'll consider it for future updates to better support web-based applications. Meanwhile, you can export the model to TensorFlow.js format using the command:

yolo export model=yolov8n.pt format=tfjs

For integrating the exported model into a browser environment, you might find the TensorFlow.js documentation helpful. Thanks for recommending Ultralytics, and we hope to meet your needs soon!

[Bharathteja07]

How to use exported engine file for inference of images in a directory?

[pderrenger]

Hey there! It looks like you're encountering an error while trying to deserialize a TensorRT engine. This error often occurs if there's a mismatch between the TensorRT version used to create the .engine file and the version you're using to load it. Here are a few steps you can try to resolve this issue:

1. Version Check: Ensure that the TensorRT version used for creating the .engine file is the same as the one you're using to load it. Version mismatches can lead to serialization errors.

2. Re-export the Model: If possible, re-export your YOLOv8 model to the TensorRT .engine format using the same environment where you're attempting to load it. This ensures compatibility.

3. TensorRT Documentation: Consult the TensorRT documentation for guidance on engine serialization and deserialization. There might be specific flags or steps required for your version.

4. Jetson Nano Deployment: For deploying on the NVIDIA Jetson Nano with JetPack 4.6.4, ensure that all components (CUDA, cuDNN, TensorRT) are correctly installed and match the versions expected by your .engine file. The NVIDIA Jetson Linux Developer Guide can be a helpful resource.

5. Sample Code: When loading and running inference with the .engine file, ensure your code correctly handles input preprocessing and output parsing specific to YOLOv8. The TensorRT Developer Guide provides examples on how to perform these steps.

If you continue facing issues, consider sharing more details about your environment and steps followed on the Ultralytics YOLOv8 GitHub Issues page. The community and developers are quite responsive and may offer more targeted advice. Good luck! 🚀

[Bharathteja07]

How can we integrate .engine model file with SAHI. Does SAHI support .engine files? my images are of size 5328*4206

[pderrenger]

Hey there! 😊 Currently, SAHI doesn't natively support .engine files directly. However, you can still use your TensorRT .engine model with SAHI by creating a custom prediction function. Here's a simplified approach:

1. Load Your Engine: Use TensorRT APIs to load your .engine file as you normally would.
2. Custom Prediction Function: Implement a function that takes an image, performs preprocessing, runs it through the TensorRT engine, and then post-processes the outputs to the format SAHI expects.
3. Integrate with SAHI: Use SAHI's get_prediction method to integrate your custom prediction function.

For large images like yours (5328x4206), consider tiling them into smaller patches that match the input size expected by your model, process each tile, and then aggregate the results.

Here's a very basic snippet to get you started:

import tensorrt as trt
import cv2
import numpy as np

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
runtime = trt.Runtime(TRT_LOGGER)

def load_engine(engine_path):
    with open(engine_path, 'rb') as f:
        engine_data = f.read()
    engine = runtime.deserialize_cuda_engine(engine_data)
    return engine

def custom_predict(image, engine):
    # Preprocess your image to match the model's expected input
    # Run inference with the engine
    # Post-process the engine's output to match SAHI's expected format
    pass

# Example usage
engine = load_engine('path/to/your/yolov8n.engine')
image = cv2.imread('path/to/your/image.jpg')
predictions = custom_predict(image, engine)

Remember, you'll need to adapt the custom_predict function to handle your specific preprocessing, inference, and post-processing steps.

For deploying on NVIDIA Jetson Nano, ensure your JetPack version is compatible with the TensorRT version used to generate the .engine file. If you encounter any issues, consider re-exporting your model with the TensorRT version available on your Jetson Nano.

Hope this helps you get started! If you have more questions or need further assistance, feel free to ask. Happy coding! 🚀

[Bharathteja07]

I am trying to run this on dGPU, I tried using ONNX with SAHI but it is taking a lot of time for processing each image.

1. As it is mentioned in the docs that ONNX (3x CPU) and Tensorrt (5X GPU), does that mean onnx doesnt use GPU? (BTW I have choosen cuda:0".
2. Could you go more in details of custom_predict function mentioned above? This I will be using with tenssort engine files

[pderrenger]

Hey there! 🌟 It looks like you're diving deep into optimizing your model's performance on a dGPU. Let's address your questions:

1. ONNX and GPU Usage: While ONNX itself is a format and doesn't dictate where the inference runs, the framework you use to run ONNX models (like ONNX Runtime) can indeed leverage GPUs. Specifying cuda:0 should direct ONNX Runtime to use your GPU. If you're experiencing slow processing, it might be worth exploring the batch size, model optimization, or the specific ONNX Runtime settings you're using.

2. Custom Predict Function Details: The custom_predict function is essentially where you'd handle the loading of your TensorRT engine, preprocessing of your input images (resizing, normalization), executing the inference, and then post-processing the outputs (decoding the detections). Since you're interested in using TensorRT .engine files, here's a simplified structure of what this function might look like:

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit  # This is needed for initializing CUDA driver

def custom_predict(image, engine):
    # Assuming 'image' is your preprocessed input image ready for the model
    # And 'engine' is your loaded TensorRT engine

    # Allocate buffers for input and output
    # [Your code to allocate buffers]

    # Create execution context
    context = engine.create_execution_context()

    # Copy input data to the buffer
    # [Your code to copy data to input buffer]

    # Run inference
    context.execute_v2(bindings=[int(d_input), int(d_output)])

    # Copy results from buffer
    # [Your code to copy data from output buffer]

    # Post-process the results to extract bounding boxes, etc.
    # [Your post-processing code]

    return processed_results

This is a very high-level overview. You'll need to fill in the details based on your specific model input/output formats and the preprocessing/postprocessing required.

For deploying on the NVIDIA Jetson Nano with JetPack 4.6.4, ensure you've installed the compatible versions of TensorRT and other dependencies. The Jetson platform is fantastic for edge AI applications, and optimizing your model with TensorRT can significantly boost inference speed.

Regarding the error you encountered, it suggests a version mismatch or an issue with the engine file itself. Double-check the TensorRT versions used for creating and loading the engine, and consider regenerating the engine file if possible.

Hope this helps clear things up! If you have more questions or need further assistance, feel free to reach out. Happy coding! 🚀

[biodatasciencearg]

I trained a custom model taking yolov8n.pt (backbone) and I want to do a model registry in MLFLOW of the model in the .engine format. It's possible directly without the export step? Someone deal with something similar? Tks for your help!

[pderrenger]

Hey there! 👋 It seems like there's a discrepancy in the class detection between your custom YOLOv8 model and the TensorRT .engine file. This usually happens due to a mismatch in the class labels or an issue during the conversion process. Here are a couple of things you might want to check:

1. Class Labels File: Ensure that the class labels file used during training is correctly referenced during the export process. The number of classes should match your custom dataset.

2. Export Command: Your export command looks good, but double-check if the model path and other parameters are correctly specified. Also, ensure that the model argument points to your custom trained .pt file.

3. TensorRT Conversion: After converting to .engine, verify that the conversion process didn't introduce any changes to the class predictions. Sometimes, the conversion process might default to a pre-set number of classes (like 91 for COCO).

Here's a simplified version of your export command for clarity:

from ultralytics import YOLO

# Load your custom trained model
model = YOLO('/content/drive/MyDrive/Colab Notebooks/people_count1/head_model.pt')

# Export to TensorRT .engine format
model.export(format='engine', imgsz=640, half=True)

If you're still facing issues, it might be helpful to explicitly specify the number of classes during the export process if the API allows, or check the documentation for any additional steps required for custom models.

Hope this helps! Let us know if you have any more questions. 😊

[sanvir19]

"I am training my custom model using the pre-trained model YOLOv8-L, which has 91 classes. When I perform inference with my custom model, it works correctly. However, after converting the custom model to TensorRT, the converted model no longer contains my custom classes; instead, it retains the 91 classes from the pre-trained model."

I am using the following code snippet to convert my model to TensorRT:
from ultralytics import YOLO
model = YOLO('custom_model.pt')
model.export(format='engine',
imgsz=640,
optimize=False,
half=True,
int8=False,
dynamic=True,
opset=None)
I have provided the correct path to the custom model.

[pderrenger]

Hey there! 👋 It looks like the issue you're encountering with the TensorRT conversion retaining the original 91 classes from the pre-trained YOLOv8-L model, instead of your custom classes, might be related to how the model's class information is handled during the export process.

When exporting your custom model to TensorRT, it's crucial to ensure that the class information from your custom dataset is correctly preserved. The discrepancy in class numbers suggests that the export process might not be correctly applying your custom model's class structure.

Here's a quick suggestion:

• Double-check that your custom dataset's class labels are correctly defined and accessible during the export process. This might involve specifying the path to your dataset's class labels file if required by the export function.

Unfortunately, the Ultralytics export function doesn't explicitly allow for specifying class labels during the export. However, the model should inherently carry over its learned classes from training. It's possible that during the conversion to TensorRT, the default COCO class labels are being assumed.

If you continue to face issues, consider reaching out on the Ultralytics GitHub issues page or forums with details of your training setup and export process. The community or the developers might offer insights specific to your scenario.

For now, ensure your custom model is correctly recognizing classes post-training and pre-export to isolate the issue to the export process. 🛠️

Hope this helps a bit! Let me know if there's anything else I can do for you.

[sanvir19]

Yes, you are correct. When I am trying to perform inference with my model, it loads its default model trained on the COCO dataset. When I found out about this issue, I provided the path to load my custom model, but then I encountered the error "jetson.inference -- detectNet failed to load network," which indicates that the model may not be loading correctly.

Could you please guide me on how to properly load the model through the entire process?

i use this code for inference:
import jetson.inference
import jetson.utils
import cv2
import time

net = jetson.inference.detectNet(argv=['--model=/home/nano/jetson-inference/data/networks/yolov8/head_mode_newl.engine',
'--labels=/home/nano/jetson-inference/data/networks/yolov8/lable.txt',
'--input-blob=input_0',
'--output-cvg=scores',
'--output-bbox=boxes'], threshold=0.5)

num_classes = net.GetNumClasses()
class_names = [net.GetClassDesc(i) for i in range(num_classes)]

[pderrenger]

@sanvir19 hey there! 👋 It looks like you're encountering an issue where your custom model, when converted to TensorRT and used with jetson.inference, defaults back to detecting the original 91 classes from the pre-trained model rather than your custom classes for head detection.

This issue might stem from how the TensorRT engine is being loaded, especially if the class labels aren't being correctly applied or recognized. Here's a suggestion to ensure your custom classes are correctly utilized:

1. Verify Class Labels: Ensure that your labels.txt file correctly lists your custom classes, and it's in the correct format expected by jetson.inference.

2. Correct Model Path: Double-check the path to your .engine file to ensure it's correctly pointing to your custom model.

3. TensorRT Conversion: When converting your model to TensorRT, ensure that the conversion process is aware of your custom classes. This might involve specifying class labels during the conversion process if supported.

Your code snippet for conversion looks good, but you might want to ensure that the --labels argument in your detectNet initialization points to the correct file and that the file is formatted correctly. Each class should be on a new line, corresponding to the class IDs your model predicts.

If you're still facing issues, consider revisiting the conversion step to ensure it's correctly tailored for your custom classes. Unfortunately, the direct integration with jetson.inference might have limitations or specific requirements for class labels and model formats, so checking the documentation for any additional steps or requirements might also help.

Hope this helps you move forward! If you have more questions or need further assistance, feel free to ask. 😊

[mrawding]

Hi, I appreciate the really awesome work within Ultralytics. I have a simple question.

What is the difference between best_int8.tflite and best_integer_quant.tflite is the quant variation simplifed more than int8 for edge devices? Curious on which one is "better" or at least the differences between the two

[pderrenger]

Hey there! Thanks for the kind words and for reaching out with your question. 😊

The best_int8.tflite and best_integer_quant.tflite files are both TensorFlow Lite models that have been quantized to use 8-bit integer weights and activations, which can reduce the model size and speed up inference, particularly on edge devices.

The main difference typically lies in the quantization technique used. best_int8.tflite usually refers to a model quantized using full integer quantization with int8 precision, while best_integer_quant.tflite might be using a different or more specific form of integer quantization.

As for which one is "better," it really depends on your specific use case and the device you're deploying to. Full integer quantization (best_int8.tflite) is generally more compatible with a wider range of devices, including those with dedicated hardware accelerators for int8 operations. However, if you're targeting a device that supports the specific quantization used in best_integer_quant.tflite, that might be the better choice for you.

In terms of simplicity, both are optimized for edge devices, but you might find one performs better than the other in your particular scenario. It's often a good idea to test both on your target device to see which gives you the best balance of speed and accuracy.

Hope that helps clear things up! If you need more detailed guidance, feel free to check out our export documentation or ask here. Happy coding! 🚀

[himanshu1027]

Hello @pderrenger

Can you plz help me out with how can i use Paddlepaddle Format to extract the text from the images?

Your response is very imp to me i am waiting for your reply.

[pderrenger]

Hello @himanshu1027,

To extract text from images using a model exported in PaddlePaddle format, you'll first need to ensure your model is trained for the task of text detection or OCR (Optical Character Recognition). Once you have a suitable model, you can export it to the PaddlePaddle format using the following code snippet:

from ultralytics import YOLO

# Load your trained model
model = YOLO('path/to/your/trained_model.pt')

# Export the model to PaddlePaddle format
model.export(format='paddle')

After exporting, you can use the PaddlePaddle inference engine to load the model and perform text detection on your images. Here's a basic example of how you might set up the inference:

import paddle
from paddle.inference import Config, create_predictor

# Setup the model
config = Config("path/to/your_model/model.pdmodel", "path/to/your_model/model.pdiparams")
config.enable_use_gpu(100, 0)
predictor = create_predictor(config)

# Prepare your image (assuming 'img' is your preprocessed image tensor)
input_names = predictor.get_input_names()
input_tensor = predictor.get_input_handle(input_names[0])
input_tensor.reshape(img.shape)
input_tensor.copy_from_cpu(img.numpy())

# Run inference
predictor.run()
output_names = predictor.get_output_names()
output_tensor = predictor.get_output_handle(output_names[0])
output_data = output_tensor.copy_to_cpu()

# Process the output_data to extract text

Make sure to adjust the preprocessing and postprocessing steps according to your specific model requirements and the format of your input data.

For more detailed guidance on using PaddlePaddle for inference, you might want to check out the PaddlePaddle documentation.

Hope this helps! Let me know if you have any more questions. 😊

[idaymand]

my code

from ultralytics import YOLO
import tensorflow as tf
import onnx2tf as onnx2tf

model = YOLO('yolov8n_web_model/yolov8n.pt') # load an official model

model = YOLO('/path_to_model/best.pt')
model.export(format="tfjs")

i got an error

ERROR: The trace log is below.
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 310, in print_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 383, in inverted_operation_enable_disable_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/ops/Conv.py", line 246, in make_node
input_tensor = get_padding_as_op(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 2031, in get_padding_as_op
return tf.pad(x, padding)
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/ubuntu/.local/lib/python3.12/site-packages/keras/src/backend/common/keras_tensor.py", line 92, in tf_tensor
raise ValueError(
ValueError: A KerasTensor cannot be used as input to a TensorFlow function. A KerasTensor is a symbolic placeholder for a shape and dtype, used when constructing Keras Functional models or Keras Functions. You can only use it as input to a Keras layer or a Keras operation (from the namespaces keras.layers and keras.operations). You are likely doing something like:

x = Input(...)
...
tf_fn(x)  # Invalid.

What you should do instead is wrap tf_fn in a layer:

class MyLayer(Layer):
    def call(self, x):
        return tf_fn(x)

x = MyLayer()(x)

ERROR: input_onnx_file_path: /home/ubuntu/Python/runs/detect/train155/weights/best.onnx
ERROR: onnx_op_name: Conv_0
ERROR: Read this and deal with it. https://github.com/PINTO0309/onnx2tf#parameter-replacement
ERROR: Alternatively, if the input OP has a dynamic dimension, use the -b or -ois option to rewrite it to a static shape and try again.
ERROR: If the input OP of ONNX before conversion is NHWC or an irregular channel arrangement other than NCHW, use the -kt or -kat option.
ERROR: Also, for models that include NonMaxSuppression in the post-processing, try the -onwdt option.

TensorFlow SavedModel: export failure ❌ 7.4s: SavedModel file does not exist at: /home/ubuntu/Python/runs/detect/train155/weights/best_saved_model/{saved_model.pbtxt|saved_model.pb}

what is wrong and what i need to do for fix?
my env
python 3.8
yolov8
tensorflow 2.16

thanks a lot

[pderrenger]

@idaymand hey there! 👋 It looks like you're encountering an error while trying to export a YOLOv8 model to TensorFlow.js (tfjs) format. The error message suggests an issue with handling KerasTensors directly in TensorFlow functions, which typically happens when attempting operations not directly supported by Keras layers or operations.

From your code snippet, it seems the error occurs during the conversion process, possibly within the onnx2tf conversion step, which is not directly related to the Ultralytics export functionality. However, to address your main concern about exporting to tfjs, here's a simplified approach using Ultralytics YOLO directly for tfjs export:

from ultralytics import YOLO

# Assuming '/path_to_model/best.pt' is your trained model
model = YOLO('/path_to_model/best.pt')  
model.export(format="tfjs") 

Make sure your environment is set up correctly with the required versions of TensorFlow and Ultralytics YOLO. Since you're using Python 3.8 and TensorFlow 2.16, the setup seems appropriate.

If you're still facing issues, consider the following steps:

1. Ensure that your model is correctly loaded before attempting the export.
2. Verify that all dependencies, especially those related to Ultralytics YOLO and TensorFlow, are correctly installed and compatible.
3. If the problem persists, you might want to explore exporting to ONNX format first and then converting to TensorFlow.js using a more direct approach or tool specifically designed for ONNX to TensorFlow.js conversion.

For more detailed guidance on exporting models and troubleshooting, the Ultralytics documentation on model export is a great resource. It covers a wide range of export formats and provides tips for optimizing your exported models.

Hope this helps! If you have further questions or run into more issues, feel free to reach out. 😊

[idaymand]

Hi @pderrenger! I see that by default ultralytics do export to onnx as step in export process yolov8 model to tfjs. I get an error on the step of export from onnx to tfjs. Log is below

my code

from ultralytics import YOLO

model = YOLO('/my_path/best.pt')
model.export(format="tfjs") # export the model to tfjs format

log

PyTorch: starting from '/home/ubuntu/Python/runs/detect/train155/weights/best.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 103, 8400) (6.7 MB)

TensorFlow SavedModel: starting export with tensorflow 2.12.0...

ONNX: starting export with onnx 1.14.0 opset 12...
ONNX: simplifying with onnxsim 0.4.35...
ONNX: export success ✅ 1.8s, saved as '/home/ubuntu/Python/runs/detect/train155/weights/best.onnx' (13.0 MB)
TensorFlow SavedModel: running 'onnx2tf -i "/home/ubuntu/Python/runs/detect/train155/weights/best.onnx" -o "/home/ubuntu/Python/runs/detect/train155/weights/best_saved_model" -nuo --non_verbose'

ERROR: The trace log is below.
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 310, in print_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 383, in inverted_operation_enable_disable_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/ops/Conv.py", line 246, in make_node
input_tensor = get_padding_as_op(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 2031, in get_padding_as_op
return tf.pad(x, padding)
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/ubuntu/.local/lib/python3.12/site-packages/keras/src/backend/common/keras_tensor.py", line 92, in tf_tensor
raise ValueError(
ValueError: A KerasTensor cannot be used as input to a TensorFlow function. A KerasTensor is a symbolic placeholder for a shape and dtype, used when constructing Keras Functional models or Keras Functions. You can only use it as input to a Keras layer or a Keras operation (from the namespaces keras.layers and keras.operations). You are likely doing something like:

x = Input(...)
...
tf_fn(x)  # Invalid.

What you should do instead is wrap tf_fn in a layer:

class MyLayer(Layer):
    def call(self, x):
        return tf_fn(x)

x = MyLayer()(x)

ERROR: input_onnx_file_path: /home/ubuntu/Python/runs/detect/train155/weights/best.onnx
ERROR: onnx_op_name: Conv_0
ERROR: Read this and deal with it. https://github.com/PINTO0309/onnx2tf#parameter-replacement
ERROR: Alternatively, if the input OP has a dynamic dimension, use the -b or -ois option to rewrite it to a static shape and try again.
ERROR: If the input OP of ONNX before conversion is NHWC or an irregular channel arrangement other than NCHW, use the -kt or -kat option.
ERROR: Also, for models that include NonMaxSuppression in the post-processing, try the -onwdt option.

TensorFlow SavedModel: export failure ❌ 5.7s: SavedModel file does not exist at: /home/ubuntu/Python/runs/detect/train155/weights/best_saved_model/{saved_model.pbtxt|saved_model.pb}

---

OSError Traceback (most recent call last)
Cell In[29], line 5
1 from ultralytics import YOLO
4 model = YOLO('/home/ubuntu/Python/runs/detect/train155/weights/best.pt')
----> 5 model.export(format="tfjs") # export the model to tfjs format

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/model.py:313, in Model.export(self, **kwargs)
311 custom = {'imgsz': self.model.args['imgsz'], 'batch': 1, 'data': None, 'verbose': False} # method defaults
312 args = {**self.overrides, **custom, **kwargs, 'mode': 'export'} # highest priority args on the right
--> 313 return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/torch/autograd/grad_mode.py:28, in _DecoratorContextManager.call..decorate_context(*args, **kwargs)
25 @functools.wraps(func)
26 def decorate_context(*args, **kwargs):
27 with self.class():
---> 28 return func(*args, **kwargs)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:261, in Exporter.call(self, model)
259 if any((saved_model, pb, tflite, edgetpu, tfjs)): # TensorFlow formats
260 self.args.int8 |= edgetpu
--> 261 f[5], keras_model = self.export_saved_model()
262 if pb or tfjs: # pb prerequisite to tfjs
263 f[6], _ = self.export_pb(keras_model=keras_model)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:122, in try_export..outer_func(*args, **kwargs)
120 except Exception as e:
121 LOGGER.info(f'{prefix} export failure ❌ {dt.t:.1f}s: {e}')
--> 122 raise e

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:117, in try_export..outer_func(*args, **kwargs)
115 try:
116 with Profile() as dt:
--> 117 f, model = inner_func(*args, **kwargs)
118 LOGGER.info(f"{prefix} export success ✅ {dt.t:.1f}s, saved as '{f}' ({file_size(f):.1f} MB)")
119 return f, model

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:712, in Exporter.export_saved_model(self, prefix)
709 for file in f.rglob('*.tflite'):
710 f.unlink() if 'quant_with_int16_act.tflite' in str(f) else self._add_tflite_metadata(file)
--> 712 return str(f), tf.saved_model.load(f, tags=None, options=None)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py:836, in load(export_dir, tags, options)
834 if isinstance(export_dir, os.PathLike):
835 export_dir = os.fspath(export_dir)
--> 836 result = load_partial(export_dir, None, tags, options)["root"]
837 return result

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py:941, in load_partial(export_dir, filters, tags, options)
936 if tags is not None and not isinstance(tags, set):
937 # Supports e.g. tags=SERVING and tags=[SERVING]. Sets aren't considered
938 # sequences for nest.flatten, so we put those through as-is.
939 tags = nest.flatten(tags)
940 saved_model_proto, debug_info = (
--> 941 loader_impl.parse_saved_model_with_debug_info(export_dir))
943 if (len(saved_model_proto.meta_graphs) == 1 and
944 saved_model_proto.meta_graphs[0].HasField("object_graph_def")):
945 metrics.IncrementReadApi(_LOAD_V2_LABEL)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/loader_impl.py:58, in parse_saved_model_with_debug_info(export_dir)
45 def parse_saved_model_with_debug_info(export_dir):
46 """Reads the savedmodel as well as the graph debug info.
47
48 Args:
(...)
56 parsed. Missing graph debug info file is fine.
57 """
---> 58 saved_model = parse_saved_model(export_dir)
60 debug_info_path = file_io.join(
61 path_helpers.get_debug_dir(export_dir),
62 constants.DEBUG_INFO_FILENAME_PB)
63 debug_info = graph_debug_info_pb2.GraphDebugInfo()

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/loader_impl.py:116, in parse_saved_model(export_dir)
114 raise IOError(f"Cannot parse file {path_to_pbtxt}: {str(e)}.")
115 else:
--> 116 raise IOError(
117 f"SavedModel file does not exist at: {export_dir}{os.path.sep}"
118 f"{{{constants.SAVED_MODEL_FILENAME_PBTXT}|"
119 f"{constants.SAVED_MODEL_FILENAME_PB}}}")

OSError: SavedModel file does not exist at: /home/ubuntu/Python/runs/detect/train155/weights/best_saved_model/{saved_model.pbtxt|saved_model.pb}

Help me please to solve this problem.
Thanks a lot

[AttilaSzucs21]

Hello I had similar problem, I downgraded my tensorflow from 2.16.1 to 2.15.1 and now it works. (In my case I tried tflite conversion so maybe it wont work in your case.)

[pderrenger]

Hey! 👋 It's great to hear that downgrading TensorFlow resolved the issue for you. TensorFlow compatibility can indeed be tricky, especially with newer versions sometimes introducing changes that affect certain functionalities.

Since our export process involves intermediate steps like converting to ONNX before reaching TensorFlow formats, there might still be discrepancies or requirements specific to those versions. If downgrading worked for your situation, it definitely might help others facing similar issues. For those still encountering errors, ensuring all conversion tools and dependencies are compatible with their TensorFlow version is crucial, as outlined in our documentation and error logs. Thanks a lot for sharing your solution! 😊

[pderrenger]

Hey there! 🌟

Thanks for sharing your experience. Downgrading TensorFlow has helped resolve compatibility issues for a lot of users, especially with model conversions. While it worked for your tflite conversion, it might indeed help in resolving OPs compatibility issues that might arise with TensorFlow.js exports as well.

If you face any more challenges, consider checking if your ONNX model uses ops supported by the version of TensorFlow you're using. Sometimes, adjusting the opset version during the ONNX export can help!

Happy coding with YOLOv8! 🚀 If you need more assistance, feel free to ask.

[Ibraheem-Ahmad]

Hello!
Would anyone be able to tell me why this code does not work:
model.export(format=='onnx', int8 = True, dyamic = True, imgsz = (width,height))

the error I get is "TypeError: Model.export() takes 1 positional argument but 2 were given"

[Aisha-Hassan1]

erro ...
wasm-core-impl.ts:219
Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'byteLength')
at t.run (wasm-core-impl.ts:219:31)
at t.run (proxy-wrapper.ts:215:17)
at t.OnnxruntimeWebAssemblySessionHandler.run (session-handler.ts:84:15)
at i.run (inference-session-impl.js:94:44)
at runModel (main.html:96:55) ........

when inputTensors => array
const inputTensors = [imageTensor1, imageTensor2];

[Aisha-Hassan1]

[batch_size, channels, height, width]. If your model expects not a single image of shape ???

[glenn-jocher]

Hey there! 👋 When converting a YOLOv8 model for batch processing, you generally don't need to specify the batch size during export, especially with dynamic batching enabled. 🔄 Your export command:

model.export(format='onnx', int8=True, dynamic=True)

is perfectly fine for creating an ONNX model that can handle arrays of images. Just ensure your data is prepped correctly when making actual predictions! 📊

For GPU acceleration with ONNX, leveraging frameworks like ONNX Runtime with CUDA support would be the way to go! 🔥 Make sure your environment is set up for GPU use, and you're specifying the execution provider accordingly.

Feel free to check out more on dynamic batching and ONNX export on the Ultralytics docs page! Hope this helps, and happy coding! 🚀

[Aisha-Hassan1]

code

const inputs = {
images: [inputTensors[0], inputTensors[1]]
};

const { output0, output1 } = await yolov8.run(inputs);

erro

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'byteLength')
at t.run (wasm-core-impl.ts:219:31)
at t.run (proxy-wrapper.ts:215:17)
at t.OnnxruntimeWebAssemblySessionHandler.run (session-handler.ts:84:15)
at i.run (inference-session-impl.js:94:44)
at runModel (main.html:107:53)

[glenn-jocher]

Hey there! 👋 It looks like there might be a slight mix-up in how you're constructing the input to your model run. When dealing with arrays of tensors in ONNX.js, you should pass a single tensor that concatenates your image tensors along the batch dimension if the model is set up for batch processing.

For your issue with inputTensors causing an error about byteLength, it's likely because the model expects one tensor that combines all input images. Instead of:

const inputs = {
    images: [inputTensors[0], inputTensors[1]]
};

You should concatenate your inputTensors into one tensor if your model expects a batch input. Unfortunately, directly passing an array like this might not work as expected with ONNX.js since it seems to expect the input as a single tensor representing the batch. The error you're encountering (TypeError: Cannot read properties of undefined (reading 'byteLength')) usually occurs when the input format to the session.run method is not as expected.

This could look different based on how your model is trained and how ort.Tensor handles batches. Ensure your model is trained with dynamic batch size if you want to process multiple images in a single forward pass. If you're not sure how to concatenate tensors in JavaScript (as direct tensor concatenation might not be as straightforward as in Python), you might have to loop over your images and process them one at a time, or look into adjusting your model to accept variable batch sizes more gracefully.

Given your current setup, if your model does not explicitly support batched inputs, you might need to process each image individually:

for(const tensor of inputTensors) {
    const inputs = { images: tensor };
    const outputs = await yolov8.run(inputs);
    // Process the outputs for each image here
}

Remember, different models expect the input in specific formats, including batch size. Ensure your preprocessing aligns with these expectations! 🚀

[JamesMcCullochDickens]

Are there any examples of getting the output of a pose estimator model in C++ using a torchscript file. I'm getting an output of shape (1, 56, 8400) for an input of size (1, 3, 640, 640) with two people in the sample picture. How should I interpret/post-process this output?

[pderrenger]

Hey there! 👋 It looks like you're diving into pose estimation with a YOLOv8 model exported as a TorchScript file. The output shape (1, 56, 8400) you're seeing is typical for pose estimation models, where 56 likely represents the number of keypoints (x, y coordinates) multiplied by the number of attributes per keypoint, and 8400 is the flattened spatial grid (for a 640x640 input, this might be 80x80 grid cells, each predicting keypoints for potential objects).

To interpret this output, you'd typically:

1. Reshape or slice the tensor to isolate keypoints for each detected person.
2. Scale the keypoints back to the original image dimensions if they're normalized.
3. Apply any necessary post-processing, like thresholding based on confidence scores if included in your output.

Here's a rough idea of how you might start to interpret the output in C++ (pseudo-code):

// Assuming output is a 1x56x8400 tensor
int num_keypoints = 28; // Example: 28 keypoints (x, y) per person
int grid_cells = 8400; // Spatial grid cells
for (int i = 0; i < grid_cells; i++) {
    // Example: Extracting the first person's keypoints
    // This part depends on how your model structures its output
    float x = output[0][i % num_keypoints * 2][i];
    float y = output[0][(i % num_keypoints * 2) + 1][i];
    // Scale x and y back to image dimensions if necessary
    // Apply any post-processing like thresholding
}

This is quite simplified and will need adjustments based on your specific model's output format. If your model includes confidence scores or other attributes per keypoint, you'll need to account for those in your parsing logic.

For more detailed guidance on exporting models and handling their outputs, our Model Export with Ultralytics YOLO page might offer some insights, even though it's more focused on the export process itself.

Hope this helps! Let us know if you have more questions. 😊

[JustinGeorgi]

I trained a yolov5 detection model a little while ago and have successfully converted that model to tensorflowjs. That tfjs model works as expected in code only slightly modified from the example available at https://github.com/zldrobit/tfjs-yolov5-example. My version of the relevant section:

const [modelWidth, modelHeight] = model.inputs[0].shape.slice(1, 3)

const input = tf.tidy(() => {
  return tf.image.resizeBilinear(tf.browser.fromPixels(imageData), [modelWidth, modelHeight]).div(255.0).expandDims(0)
})

var results = model.executeAsync(input).then(res => {
  const [boxes, scores, classes, valid_detections] = res
  const boxes_data = boxes.dataSync()
  const scores_data = scores.dataSync()
  const classes_data = classes.dataSync()
  const valid_detections_data = valid_detections.dataSync()[0]

I have now trained a yolov8 detection model on very similar data. The comments in exporter.py indicate that in that example script v8 models should be a drop in replacement for v5 models:

https://github.com/ultralytics/ultralytics/blob/main/ultralytics/engine/exporter.py#L45-L49

However, that does not seem to be the case. The v5 model output is the 4 length array of tensors (which is why the destructuring assignment works), but the v8 model output is a single tensor of shape [1, X, 8400] thus the example code results in an error complaining that the model result is non-iterable when attempting to destructure.

From what I understand, the [1, X, 8400] is the expected output shape of the v8 model. Is further processing of the v8 model required, or did I do something wrong during the pt -> tfjs export?

[pderrenger]

Hey there! 😊 It looks like you've encountered a difference in output formats between YOLOv5 and YOLOv8 models when exporting to TensorFlow.js. Indeed, YOLOv8 models output a single tensor of shape [1, X, 8400], which is different from the 4-length array of tensors you'd get from YOLOv5 models.

For YOLOv8, you'll need to process the single output tensor to extract the bounding boxes, scores, and class IDs. This involves reshaping and slicing the tensor based on your model's specific output format. Here's a simplified example of how you might start to approach this:

model.executeAsync(input).then(res => {
  // Assuming res is the single tensor output [1, X, 8400]
  const data = res.dataSync();
  
  // Example processing (you'll need to adjust based on your model specifics)
  const numDetections = data[0];
  for (let i = 0; i < numDetections; i++) {
    const offset = i * 7; // Example offset, adjust based on your output format
    const score = data[offset + 2]; // Example index for score
    // Extract other values (x, y, width, height, class) similarly
  }
  // Further processing...
});

This is a very general example, and you'll need to adjust the indices based on how your YOLOv8 model formats its output tensor. The key takeaway is that the output tensor will contain all the information you need, but it will require some additional processing to extract and use effectively.

For more detailed guidance on handling YOLOv8's output format and integrating it with TensorFlow.js, I recommend checking out our Model Export documentation, which provides insights into exporting models and might offer further clues on handling different output formats.

If you have any more questions or need further assistance, feel free to ask. Happy coding! 🚀

[JustinGeorgi]

Thank you, for the reply. I thought reshaping and slicing should be something like that, but was concerned because that doesn't make any sense with the output that I'm getting.

For example:

const numDetections = data[0];

implies that, after reshaping the results, data[0] should be an integer value. As should any values that refer to class indicies of the detected objects. There are no integer values in the result array.

I know the original pt model works, it can correctly identify objects when I do a quick yolo predict ... command line test. So now, I geuss, that leaves me with two options: the pt -> tfjs has gone wrong or my input value for executeAsync is incorrect. I didn't set any additional preprocessing parameters on the origin model build, so the only things my js code does is reshape the image to the size defined in the model (which is correct) and normalize the data.

Do the yolov8 models have any additional pre-processing that I am unaware of or is it most likely that my exported model is faulty?

[pderrenger]

Hey there! 👋 It sounds like you're diving deep into the intricacies of model outputs and preprocessing, which is fantastic! From what you've described, it seems there might be a bit of confusion regarding the output format of YOLOv8 models and how to handle them in TensorFlow.js.

Indeed, YOLOv8 models output a single tensor with a shape like [1, X, 8400], which is quite different from YOLOv5's multi-tensor output. This single tensor packs all the detection information, including bounding boxes, objectness scores, and class probabilities, in a compact form. The key to using this output effectively is to correctly parse and reshape this tensor to extract the information you need.

Regarding preprocessing, YOLOv8 models generally follow the same preprocessing steps as YOLOv5 models: resizing the input image to the model's input dimensions and normalizing pixel values to the range [0, 1]. If you've got that covered, your preprocessing should be on point! 🎯

If the output tensor doesn't seem to contain integer values where you expect them (like class indices), it might be due to how the data is packed in the tensor. The actual values you're looking for are there but might require a different approach to extract correctly.

Given your situation, it doesn't immediately sound like a fault in the pt -> tfjs conversion process, especially if the model works as expected in a yolo predict test. It's more likely about adapting the post-processing step to match YOLOv8's output format.

For a more tailored solution, I'd recommend revisiting the Model Export documentation and focusing on the specifics of handling YOLOv8's output tensor in TensorFlow.js. Unfortunately, without a direct example of parsing YOLOv8's output in TensorFlow.js, I can't provide a precise code snippet right now. However, understanding the structure of the output tensor and applying the correct tensor operations to unpack it will be key.

Keep experimenting, and don't hesitate to reach out if you hit another snag or have more questions. Happy coding! 😄

[Ibraheem-Ahmad]

I was wondering if anyone could help me with this code: I exported my custom trained yolov8n.pt model to .onnx but now my code is not working(model.export(format='onnx', int8=True, dynamic=True)). I am having trouble using the outputs after running inference.

My Code:
"import cv2
import onnxruntime as ort
import numpy as np

def load_image(image_path):
# Load the image from the specified path
original_image = cv2.imread(image_path)
if original_image is None:
raise ValueError(f"Could not load image at path: {image_path}")

# Resize the image to the required dimensions (640x640)
resized_image = cv2.resize(original_image, (640, 640))

# Rearrange the image dimensions from HWC to CHW format and add a batch dimension
image_data = np.transpose(resized_image, (2, 0, 1))
image_data = np.expand_dims(image_data, axis=0).astype(np.float32)

return original_image, image_data

def draw_bounding_boxes(image, detections, confidence_threshold=0.5):
for detection in detections:
x1, y1, x2, y2, score = detection
if score >= confidence_threshold:
start_point = (int(x1), int(y1))
end_point = (int(x2), int(y2))
color = (255, 0, 0) # Bounding box color
thickness = 2 # Bounding box thickness
image = cv2.rectangle(image, start_point, end_point, color, thickness)
return image

def main(model_path, image_path):
# Load ONNX model
ort_session = ort.InferenceSession(model_path)
input_name = ort_session.get_inputs()[0].name
input_shape = ort_session.get_inputs()[0].shape

# Load and preprocess the image
original_image, image_data = load_image(image_path)

# Run inference
detections = ort_session.run(None, {input_name: image_data})[0]

for output in detections[0]:
    print(f"Output {output}: Shape = {output.shape}, Type = {output.dtype}")
    # Print values for smaller tensors
    if output.size < 500:  # Arbitrary size threshold to avoid flooding the console
        print(f"Values:\n{output}")
# Assume detections are in the format [batch, num_detections, 6] where each detection is [x1, y1, x2, y2, score, class]
detections = detections # Assuming batch size of 1 for simplicity


# Draw bounding boxes on the original image
final_image = draw_bounding_boxes(original_image, detections)

# Save or display the image
#cv2.imwrite('output.jpg', final_image)
cv2.imshow('Detected Objects', final_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

if name == "main":
model_path = 'weights\best.onnx' # Replace with your model path
image_path = "Capture.PNG" # Replace with your image path
main(model_path, image_path)
"

Error:
line 28, in draw_bounding_boxes
if score >= confidence_threshold:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

[glenn-jocher]

Hey there! It looks like you're encountering an issue with handling the output from your ONNX model after running inference. The error you're seeing is likely due to attempting to directly compare a numpy array with a scalar value, which numpy doesn't allow without specifying whether you want to use any() or all() for the comparison.

However, the root cause seems to be how you're processing the detections output from the ONNX model. Typically, the output from an ONNX model will be a numpy array, but you need to ensure you're accessing the detection results correctly. Here's a simplified approach to process the model output and draw bounding boxes:

# Assuming detections are in the format [batch, num_detections, 6] where each detection is [x1, y1, x2, y2, score, class]
# And assuming batch size of 1 for simplicity
detections = detections[0][0]  # Adjust based on your model's output format

# Draw bounding boxes on the original image
final_image = draw_bounding_boxes(original_image, detections, confidence_threshold=0.5)

Make sure the detections variable is correctly formatted before passing it to draw_bounding_boxes. The error you're encountering suggests that detections might not be unpacked correctly, leading to the comparison issue.

If you're still having trouble, could you share more details about the shape and type of the detections output right after inference? That might help pinpoint the issue further. 😊

[larryKra]

"batch_size" is not in arguments as previous versions?

[glenn-jocher]

Hey there! 👋

In YOLOv8, the batch_size parameter is typically set within your dataset YAML file rather than as a direct argument in training commands. This allows for more flexibility and dataset-specific batch size configurations. Here's a quick example of how it's defined in a dataset YAML:

# Example dataset.yaml
train: /path/to/train/images
val: /path/to/val/images

# Define your batch size here
batch_size: 16
...

Then, when you run your training command, YOLOv8 will automatically pick up the batch_size from your dataset configuration. No need to pass it as a separate argument! 😊

Hope this helps! If you have any more questions, feel free to ask.

[yasirgultak]

I converted the model I trained with costum data to tflite format. Before converting, I set the int8 argument to true. But when I examined the tflite format from the netron website, I saw that the input information is still float32. Is this normal or is there a bug?

Also thank you very much for answering every question without getting bored.

[glenn-jocher]

Hey there! 😊 It's great to hear you're exploring the TFLite format for your custom model. Regarding your observation about the input information still being float32 even after setting int8 to true, this is actually expected behavior. The int8 quantization primarily affects the weights and activations within the model to reduce size and potentially speed up inference on compatible hardware. However, the input and output layers often remain as float32 for compatibility reasons, allowing you to feed in standard floating-point data and receive floating-point outputs without additional conversion steps.

If you're looking to further optimize or have specific requirements for input/output data types, you might consider post-processing steps or exploring additional TensorFlow tools for model optimization.

And no worries at all, we're here to help! If you have any more questions or need further assistance, feel free to ask. Happy coding! 🚀

[usama1123456789]

!yolo export model=/content/drive/MyDrive/best-1-1.pt format=tflite

export failure ❌ 33.0s: generic_type: cannot initialize type "StatusCode": an object with that name is already defined
Traceback (most recent call last):
File "/usr/local/bin/yolo", line 8, in
can anyone help me to export to tflite....

[usama1123456789]

Firstly, a big thank you to @glenn-jocher for the support.

During my search for a solution, I discovered that TensorFlow versions greater than 2.13.0 have issues with YOLO.

As Google Colab had version 2.15.0 installed, I resolved the issue by adding the following snippet before the conversion:

%pip install tensorflow==2.13.0
This ensured compatibility and resolved the problem. Thank you again for the assistance!

[glenn-jocher]

Hey there! 🎉 Big thanks for sharing your experience and solution - it's super valuable! 🙌 It's really interesting to hear about the version-specific intricacies with TensorFlow and YOLO. Downgrading TensorFlow to 2.13.0 to ensure compatibility is a clever workaround. Your discovery is a great addition to the community knowledge pool, and it might just be the key for others facing similar issues. 👍

If anyone else encounters similar hiccups, trying out different TensorFlow versions based on your model requirements could indeed be a fruitful debugging step.

Super grateful for your input, and cheers for contributing to smoother YOLO model deployments! 🚀 If there's more to ponder or if new challenges pop up, the community is here to help. Keep up the fantastic work!

[usama1123456789]

yolo export model=/content/yolov8nsss (3).pt format=tflite

/bin/bash: -c: line 1: syntax error near unexpected token (' /bin/bash: -c: line 1: yolo export model=/content/yolov8nsss (3).pt format=tflite'

Hi i am converting Yolov8n file to tflite ia am facing this issue suggest me solution
thank you

[eladoni1]

Isnt that because your file name? I do not use the CLI, but try to add " (quotes) in the parameters, or, change the file name to something without paranthesys.

So, either :
yolo export model="/content/yolov8nsss (3).pt" format="tflite"

Or, if that doesnt work, rename :
yolov8nsss (3).pt

To :

yolov8nsss.pt

(Just without paranthesys).

[glenn-jocher]

@eladoni1 hey! Great catch on figuring out the solution with the file name! 🌟 Yes, file paths with special characters, including spaces and parentheses, can sometimes throw a wrench into command line processing. Wrapping the path in quotes or simplifying the file name are both solid approaches.

For others who might come across similar issues, here's a quick recap based on the awesome tip shared:

To handle file names with spaces or special characters when using the command line, you can either:

• Wrap the path in quotes, like so:

yolo export model="/content/yolov8nsss (3).pt" format="tflite"

• Or, rename the file to remove spaces and special characters, making it:

yolov8nsss.pt

Both methods help ensure the command line correctly understands the file path, aiding in a smooth export process. 🚀 Thanks again for highlighting this solution, and if there's anything more you're curious about or need help with, feel free to ask!

[rajazain2001]

Hi I havr tried all TFLITE export formats to convert the best.pt to .tflite but non is working. I have also checked my runtime and all the latest imports pip install -U ultralytics, and I have also tried the code you gave to someone in the comments but the issue is not resolvig

Step 1: Export to TensorFlow SavedModel

!yolo export model='/content/drive/MyDrive/best-1-1.pt' format=saved_model

Step 2: Convert the exported SavedModel to TensorFlow Lite

import tensorflow as tf
saved_model_dir = 'path/to/exported/saved_model' # Update this path
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()

Save the TFLite model

with open('/content/drive/MyDrive/yolov8_model.tflite', 'wb') as f:
f.write(tflite_model) .

but the same error comes back.
can you please provide me with some specific solution I wil be very thankful to you.
you can reach me at zainayani2001@gmail.com

[pderrenger]

@rajazain2001 hi there,

It sounds like you're encountering issues while trying to convert your best.pt model to TensorFlow Lite format. Let's try a couple of things to troubleshoot:

1. Ensure Compatibility: Make sure that your model architecture and operations are compatible with TFLite. Some custom layers or operations might not be supported directly.

2. Update TensorFlow: Sometimes, updating TensorFlow to the latest version can resolve conversion issues:

   pip install --upgrade tensorflow

3. Use TensorFlow 2.x: Ensure you are using TensorFlow 2.x, as TensorFlow 1.x might not support some functionalities.

4. Check the Error Message: The specific error message you're receiving can give more clues on what might be going wrong.

5. Simplify the Model: If possible, try simplifying the model or using a standard model to test if the issue is with the model complexity.

If these steps don't resolve the issue, could you please share the specific error message you're encountering? This will help in diagnosing the problem more effectively.

For more detailed guidance on exporting models, you might find our export documentation helpful.

[itachi1232gg]

can we export sam/mobile sam model to tensorRT or onnx?

[pderrenger]

Absolutely! You can export SAM/MobileSAM models to both TensorRT and ONNX formats using the Ultralytics YOLOv8 export functionality. Here's how you can do it:

For ONNX:

yolo export model=path/to/your/sam_model.pt format=onnx

For TensorRT:

yolo export model=path/to/your/sam_model.pt format=engine

Just replace path/to/your/sam_model.pt with the actual path to your trained SAM or MobileSAM model. Let me know if you need further assistance! 😊

[itachi1232gg]

Thanks for answering, I tried yolo export model=path/to/your/sam_model.pt format=engine , it raises error : AttributeError: 'Sam' object has no attribute 'args'. SAM and YOLO extend from different base class.

[pderrenger]

Hey there! 👋 Thanks for reaching out with your issue. It seems like you've encountered an attribute error concerning the SAM model during export to TensorRT. This error suggests the SAM class is missing an expected attribute, likely due to differences in the base class structures between SAM and YOLO models.

A potential workaround could involve ensuring that your SAM model object has all necessary attributes before export. However, without diving deeper into the codebase, it's challenging to provide an exact solution.

Could you try exporting the model to ONNX first and then converting that ONNX file to TensorRT? This two-step process might bypass the immediate issue:

# Export to ONNX
yolo export model=path/to/your/sam_model.pt format=onnx

# Convert ONNX to TensorRT (using an external tool or script)

If this doesn't resolve the issue, could you provide more details about your SAM model and the environment you're working in? That could give us better insight into the problem. 🛠️

Also, we're continuously improving our models and export capabilities. So, your feedback is invaluable for making YOLOv8 even better!

[itachi1232gg]

I think exporter only supports yolo models. In order to convert SAM model to ONNX or TensorRT, I should write anther convert script.

[pderrenger]

Hey! 👋 Thanks for reaching out. Yes, the current exporter is primarily designed for YOLO models. However, for SAM/MobileSAM models, converting to ONNX or TensorRT does indeed require a bit of custom handling.

To get started with ONNX export, you might consider something along these lines after your model is loaded:

import torch.onnx

# Assuming `model` is your SAM/MobileSAM model
torch.onnx.export(model, dummy_input, 'model_name.onnx', verbose=True, input_names=['input'], output_names=['output'])

This code snippet assumes a dummy input that matches the expected dimensions of the model.

For TensorRT, after exporting to ONNX, tools like onnx-tensorrt can help with the conversion:

onnx2trt model_name.onnx -o model_name.trt

Although these steps are a good starting point, depending on the SAM model's specifics, adjustments might be necessary. If you encounter challenges along the way or need further assistance, feel free to ask! We're here to help. 😊

[Synco9]

Hello!
First thing of all, I appreciate the really awesome work in Yolov8.
But I'm facing some problems with the version of CUDA==12.4, TensorRT==10.0.0b6. When I was exporting with simple script:

model.export(format='engine')
and it keeps telling me that:
'tensorrt.tensorrt.IBuilderConfig' object has no attribute 'max_workspace_size'
Could you guys help me solved that?

[pderrenger]

@Synco9 hey there! 🌟 Thank you so much for the kind words about YOLOv8 and for bringing this issue to our attention. It looks like you've encountered a compatibility issue with TensorRT 10.0.0b6. The attribute max_workspace_size seems to be missing or renamed in this version.

First, could you please try updating your TensorRT installation to a stable release version, if possible? Beta versions can sometimes introduce unexpected changes or issues.

If you're bound to using TensorRT 10.0.0b6, as a workaround, you might need to modify the export script slightly to accommodate the changes in this TensorRT version. Unfortunately, without direct access to this specific beta version, I can't provide precise code modification. However, looking into the TensorRT documentation for changes related to the IBuilderConfig and how the workspace size is set in this version might offer some clues.

In the meantime, if anyone has faced and solved this issue, feel free to jump in! Your contributions are greatly appreciated. 🚀

If you have further questions or need more assistance, don't hesitate to ask.

[Synco9]

@pderrenger hi there! After re-updating my TensorRT to version 8 it worked just perfectly. Apparently they changed the names of many fields after version 10 of TensorRT. Since my recent work is related to this area, that's why this question arose.
Thank you very much for your reply! If you guys find out more or have some update of this version of TensorRT, please let us know as soon as possible!

[pderrenger]

@Synco9 Hi there! 🎉 Great to hear that reverting to TensorRT version 8 resolved the issue for you. Indeed, newer versions can sometimes change API behaviors unexpectedly. We'll keep an eye on updates and changes in the newer versions of TensorRT and will update our documentation and support threads accordingly. If there's anything new or significant, we'll make sure to share it with the community. Thanks for keeping us in the loop, and happy coding! 🚀

[Hariii17]

I have trained my model and have downloaded it now i want to deploy my model in an webinterface for detecting object in live video feed can you provide me with the webinterface code

[glenn-jocher]

Hello! 🚀 I'm really excited to hear that you're looking to deploy your model in a web interface for live video object detection. While we don't have a specific web interface code ready out of the box, deploying trained YOLOv8 models in a web application is definitely feasible!

Here's a simplified roadmap to get you started:

1. Export Your Model: If you haven't done so, export your YOLOv8 model to ONNX for broader compatibility. For example:

   yolo export model=path/to/your/model.pt format=onnx

2. Use ONNX with a Server Backend: Integrate the ONNX model with a backend server using Flask or FastAPI. This backend can handle incoming video streams, process them frame-by-frame, and return the detection results.
3. Web Frontend: Develop a simple frontend using HTML/JavaScript to capture live video streams from the browser and send it to your backend for processing.

For deploying models, especially for live video feeds, consider using WebSockets for a more seamless and real-time interaction. There are also several great resources and tutorials out there for integrating ONNX models with Flask or FastAPI and connecting them to a frontend. Unfortunately, we can't provide the complete code, but this roadmap should give you a solid start.

Remember, deploying models to a live environment requires tuning for performance and resource management, so make sure to adjust your model's input size and experiment with quantization if needed for better speed.

Happy coding, and we look forward to seeing your web interface in action! 🌐

[AlixYehia]

Hi, I have an issue with using a custom yolov8 model "best.pt" since i was using yolov3 but i wanted to upgrade to yolov8 the following code is how i used the old model

import cv2
import numpy as np
from tracker import Tracker
import cvzone
import os
from datetime import datetime
from pynput import mouse

def get_coords(x, y):
    print("Now at: {}".format((x, y)))

# Start the mouse listener
with mouse.Listener(on_move=get_coords) as listener:
    # Load Yolo
    net = cv2.dnn.readNet("yolov3_training_last.weights", "yolov3_testing.cfg")

    # Name custom object
    classes = ["car"]

    # Initialize the video capture
    cap = cv2.VideoCapture(r"C:\Users\aliye\OneDrive\Pictures\Saved Pictures\IMG_8189.MOV")  # Replace 'your_video.mp4' with your video file path

    # Create a background subtractor
    fgbg = cv2.createBackgroundSubtractorMOG2()
    area1 = [(360, 300), (360, 400), (600, 400), (600, 300)]
    area2 = [(360, 175), (360, 275), (600, 275), (600, 175)]
    tracker = Tracker()
    a1 = {}
    counter = []

    def save_full_frame(frame):
        # Create a folder with the current date and time as the folder name
        current_datetime = datetime.now().strftime("%Y%m%d%H%M%S")
        folder_name = f"wrongway"
        os.makedirs(folder_name, exist_ok=True)

        # Save the entire frame
        image_filename = os.path.join(folder_name, f"frame_{current_datetime}.jpg")
        cv2.imwrite(image_filename, frame)

    # Initialize variables for frame delay and pause state
    delay = 1  # Adjust the delay value (in milliseconds) to control the playback speed
    is_paused = False

    layer_names = net.getLayerNames()
    output_layers = [layer_names[i - 1] for i in net.getUnconnectedOutLayers()]
    colors = np.random.uniform(0, 255, size=(len(classes), 3))

    while True:
        if not is_paused:
            ret, frame = cap.read()

            if not ret:
                break
            frame = cv2.resize(frame, (1020, 500))
            height, width, channels = frame.shape  # Move this line here

            # Apply background subtraction
            fgmask = fgbg.apply(frame)

            # Threshold the foreground mask
            _, thresh = cv2.threshold(fgmask, 250, 255, cv2.THRESH_BINARY)

            # Find contours in the thresholded image
            contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

            # Draw rectangles around moving objects
            list = []
            for contour in contours:
                if cv2.contourArea(contour) > 1000:  # Adjust the area threshold as needed
                    x, y, w, h = cv2.boundingRect(contour)
                    list.append([x, y, w, h])
            bbox_idx = tracker.update(list)
            for bbox in bbox_idx:
                x1, y1, w1, h1, id = bbox
                cx = int(x1 + x1 + w1) // 2
                cy = int(y1 + y1 + h1) // 2
                result = cv2.pointPolygonTest(np.array(area1, np.int32), ((cx, cy)), False)
                if result >= 0:
                    a1[id] = (cx, cy)
                if id in a1:
                    result1 = cv2.pointPolygonTest(np.array(area2, np.int32), ((cx, cy)), False)
                    if result1 >= 0:
                        cv2.rectangle(frame, (x1, y1), (x1 + w1, y1 + h1), (0, 255, 0), 2)
                        cv2.circle(frame, (cx, cy), 6, (0, 255, 0), -1)
                        cvzone.putTextRect(frame, f'{id}', (x1, y1), 1, 1)
                        if counter.count(id) == 0:
                            counter.append(id)
                            save_full_frame(frame)

            # Object detection with YOLO
            blob = cv2.dnn.blobFromImage(frame, 0.00392, (416, 416), (0, 0, 0), True, crop=False)
            net.setInput(blob)
            outs = net.forward(output_layers)

            # Showing informations on the screen
            class_ids = []
            confidences = []
            boxes = []
            for out in outs:
                for detection in out:
                    scores = detection[5:]
                    class_id = np.argmax(scores)
                    confidence = scores[class_id]
                    if confidence > 0.3:
                        center_x = int(detection[0] * width)
                        center_y = int(detection[1] * height)
                        w = int(detection[2] * width)
                        h = int(detection[3] * height)

                        # Rectangle coordinates
                        x = int(center_x - w / 2)
                        y = int(center_y - h / 2)

                        boxes.append([x, y, w, h])
                        confidences.append(float(confidence))
                        class_ids.append(class_id)

            indexes = cv2.dnn.NMSBoxes(boxes, confidences, 0.5, 0.4)
            font = cv2.FONT_HERSHEY_PLAIN
            for i in range(len(boxes)):
                if i in indexes:
                    x, y, w, h = boxes[i]
                    label = str(classes[class_ids[i]])
                    cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
                    cv2.putText(frame, label, (x, y - 7), font, 1, (0, 255, 0), 1)

            cv2.polylines(frame, [np.array(area1, np.int32)], True, (0, 255, 0), 2)
            cv2.polylines(frame, [np.array(area2, np.int32)], True, (0, 0, 255), 2)
            p = len(counter)
            cvzone.putTextRect(frame, f'WrongsideVehicle:-{p}', (50, 60), 2, 2)
            cv2.imshow('Motion Detection', frame)

        key = cv2.waitKey(delay) & 0xFF
        if key == 27:  # Press 'Esc' to exit
            break
        elif key == ord(' '):  # Press spacebar to pause/resume
            is_paused = not is_paused

    # Release the video capture and close all OpenCV windows
    cap.release()
    cv2.destroyAllWindows

so the line
net = cv2.dnn.readNet("yolov3_training_last.weights", "yolov3_testing.cfg")

I have a .cfg file but for yolov8 i only have my weight and i tried many things but I cant figure out how to use the new model with that code. sorry since I am new in this

[glenn-jocher]

Hey there! 👋 It sounds like you're transitioning from YOLOv3 to YOLOv8 and looking to run inference with your custom best.pt model. It's great to see your enthusiasm for upgrading! Here's a simplified way to use your YOLOv8 model for inference:

from ultralytics import YOLO

# Load your custom trained model
model = YOLO('path/to/best.pt')

# Load your image with OpenCV
frame = cv2.imread("path/to/your/image.jpg")

# Run inference
results = model(frame)

# Results contain detections, which you can then display or save. Here's how to plot:
annotated_frame = results[0].plot()

# Show the annotated image
cv2.imshow("YOLOv8 Inference", annotated_frame)
cv2.waitKey(0)  # Wait for a key press to close the displayed window

Please replace "path/to/your/image.jpg" and "path/to/best.pt" with the actual paths to your image and model. This example assumes you're running inference on a single image. For videos or live camera feeds, you would loop through frames and call model() on each frame, similar to your YOLOv3 setup.

If you're using a video file, remember to read the frame inside the loop like you did previously with cap.read(). The results object has a lot of useful attributes and methods, such as .show() method to display the results directly. For further customization and advanced features, I highly recommend checking our documentation for more details.

Hope this helps you get started with YOLOv8! Let me know if you have any more questions. 😊

[eladoni1]

Exporting your default 'yolov8s-cls.pt' (not just small, any size to be exact), yields different results completely.

Does TensorFlow models do different post-processing then the same model after export to (lets say) ONNX? and to OpenVino?

Given a bus image, I am given 0.67 confidence it's a trolleybus, while the exported has given me 0.51 confidence..

What am I doing wrong? post processing? export variables? or the results are bound to change and nothing can be done?

Thanks.

[eladoni1]

edit - I have been using yolo's model reader (I mean, YOLO(%modelPath%)), for the following models -
TensorFlow - .pt file (default model)
ONNX - .onnx file
OpenVino - folder for xml, bin and yaml

ONNX and OpenVino gave same results. but not the same as .pt file (When image dimensions were 1184x789)

When picture dimensions were changed to 224x224 the results were the same (But now the w/h ratio of the image is changed drastically).

[glenn-jocher]

@eladoni1 hi there! Thanks for sharing your observations regarding the model export and the results discrepancies you've encountered across different formats and image dimensions. 🌟

It's usual for there to be slight variations in confidence scores or inference outputs between different model formats (e.g., .pt, .onnx, and OpenVINO). This can stem from a few reasons:

1. Quantization and Precision: Going from PyTorch to ONNX or OpenVINO, especially when using half-precision (FP16) or quantization (INT8), can lead to slight precision losses. While these techniques optimize inference speed and model size, they can slightly alter the outputs.

2. Resolution Changes: The change in confidence scores with different image resolutions (1184x789 vs. 224x224) hints at the importance of maintaining aspect ratios or using the same preprocessing steps used during training. Resizing can sometimes distort images, influencing model performance.

3. Post-processing Differences: Each ecosystem (PyTorch vs. ONNX vs. OpenVINO) has its nuances in how post-processing (like NMS) is implemented or optimized. These differences might affect the final confidence scores.

If you're observing significant discrepancies, here are a few checks you might want to consider:

• Ensure that the preprocessing and post-processing steps (such as normalization, resizing methods, and NMS) are consistent across all formats.
• When exporting, try to preserve the original aspect ratio of your images, or apply the same preprocessing as during training.
• Revisit the export settings. Parameters like dynamic, half, or simplify might be adjusted to closer match the original .pt model's behavior.

For example, when exporting to ONNX with dynamic axes and keeping the original aspect ratio, your code might look something like:

yolo export model=yolov8s-cls.pt format=onnx imgsz=1184,789 dynamic=True

And remember, slight variations in output scores are expected due to the reasons mentioned above, but these should generally not drastically change the model's performance or conclusions.

Let's keep the discussion going if you have more insights or specific challenges! 🚀

[eladoni1]

Hi glenn, thank you for your fast response~
couple of follow up questions -
A) The fact that the task is actually classification rather than detect, doesn't it remove the possibility it is a post-processing problems? (Like NMS is only in detection)

B) Considering I am using only YOLO for the loading of the models and the prediction of an image (task = classify), does YOLO handle pre-processing entirely? or when I load the exported model and do the prediction, I should actually divide by 255 myself, and resize the image to a w:h ratio of 1:1 myself?
Also on that topic - where can I see all of the normalization / augmentation done to the data done before the first layer? or actually see the image we are passing after it's preprocess?

C) Considering we are NOT using any of the quantizations or reduced accuracy, what is considered as 'slight' variations in confidence scores? because the results are somewhat relevant, but reduced the overall top1 confidence by more than 10%.

D) New topic - Unrelated to previous questions - When you said that YOLO handle resizing the image to fit the imgsz argument (but keeps the w/h ratio ofcourse), does it fill the void by black pixels? or gray?

Again, much appreciated glenn

[glenn-jocher]

@eladoni1 hi there! Thanks for the follow-up questions, and I'm glad to dive into these details for you. 🚀

A) You're correct that for classification tasks, concerns like NMS don't apply since we're not dealing with object localization. Variations are more likely related to preprocessing or the model export process itself.

B) When using the YOLO models through our API, we handle preprocessing steps automatically, including resizing and normalizing images. If you're curious or need to replicate the preprocessing pipeline manually, here's a quick glance at the usual steps:

# Example for resizing and normalizing
transform = transforms.Compose([
    transforms.Resize((640, 640)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

Normalization values (mean and std) and other preprocessing steps are typically consistent across model training and inference to maintain performance. You can see the exact transformations in the training script or configuration.

C) While 'slight' variations can be subjective, a more than 10% difference in top-1 confidence might indicate needing a closer look at preprocessing, as before mentioned, or export parameters. It's worth experimenting with export settings or ensuring the input to your model matches training conditions as closely as possible.

D) When resizing images while keeping the aspect ratio, additional padding is indeed applied. The default behavior fills with black (0,0,0) pixels. If you're working directly with the PyTorch transformations or another toolkit, this can usually be customized.

Hope this helps clarify things a bit! If you have more questions or need further assistance, feel free to ask. Cheers! 🌟

[hammockrobotics]

is there a way to export the model in tensorRT format from .pt file in the varying precision's: INT8, FP16, FP32, and full precision FP64 ??

[youssefoumate]

are these warnings normal during the export of the yolov8n model (tflite, int8):
Model conversion started ============================================================
WARNING: The optimization process for shape estimation is skipped because it contains OPs that cannot be inferred by the standard onnxruntime.
WARNING: module 'onnx' has no attribute '_serialize'

[glenn-jocher]

Hey there! 👋

Yes, during model export, especially when targeting optimized formats like tflite with quantization (int8), it is common to encounter warnings regarding shape estimation and certain OPs not being inferred by the onnxruntime. This generally occurs due to the optimization processes trying to adapt the model to a format that's highly optimized for inference speed and size. As for the warning regarding the 'onnx' attribute _serialize, it might indicate a version mismatch or an internal optimization step that does not impact the final export negatively.

It's a good practice to ensure your environment is up to date and compatible with the export process requirements. However, these warnings often don't impede the export process significantly but always good to check your exported model's performance and accuracy.

Feel free to share your model's performance post-export, and if you encounter any errors or noticeable degradation in model accuracy, we're here to help! 😊

[sh3mmy]

i've trained my model and its detecting pretty well in python, so now i want to use Embedded C with ML Libraries or an alternative to imbed my code into a stm32. I'm using a realsense 3d camera, and this is my python code: import cv2
from ultralytics import YOLO
import numpy as np
import pyrealsense2 as rs

Load YOLOv8 model

model = YOLO("best.pt")

Configure RealSense pipeline

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)

Start streaming

pipeline.start(config)

try:
while True:
frames = pipeline.wait_for_frames()
color_frame = frames.get_color_frame()
depth_frame = frames.get_depth_frame()
if not color_frame or not depth_frame:
continue

    # Convert color frame to numpy array
    image = np.asanyarray(color_frame.get_data())

    # Run YOLOv8 inference
    results = model(image, conf=0.6)

    # Process detections
    for det, conf_v, cls_v in zip(results[0].boxes.xyxy, results[0].boxes.conf, results[0].boxes.cls):
        xmin, ymin, xmax, ymax = det
        confidence = conf_v
        classes = cls_v

        # Draw bounding box
        cv2.rectangle(image, (int(xmin), int(ymin)), (int(xmax), int(ymax)), (0, 0, 255), 2)

        # Calculate centroid
        centroid_x = int((xmin + xmax) / 2)
        centroid_y = int((ymin + ymax) / 2)

        # Get depth at centroid
        depth = depth_frame.get_distance(centroid_x, centroid_y)

        # Convert depth to real world coordinates
        depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
        xyz = rs.rs2_deproject_pixel_to_point(depth_intrin, [centroid_x, centroid_y], depth)

        print(f"Confidence: {confidence:.1f}")
        print(f"Class_val: {classes:.1f}")
        print(f"Centroid XYZ: {xyz}, Depth: {depth:.2f} meters")

    # Display the resulting frame
    cv2.imshow('Object Detection', image)
    cv2.waitKey(1)  # Add this line to allow the OpenCV window to refresh

finally:
pipeline.stop()
cv2.destroyAllWindows()

how do i convert it to TensorFlow/ tensorflow lite

[pderrenger]

Hey there! 🎉 It sounds like you've made some great progress with your YOLOv8 model and are ready for the exciting step of embedding it into an STM32. For converting your model to TensorFlow or TensorFlow Lite, here's what you can do:

1. Export your model to TensorFlow SavedModel or TensorFlow Lite format using Ultralytics YOLO's export functionality. Here's a quick example to get you started:

   yolo export model=best.pt format=saved_model  # For TensorFlow SavedModel

   Or for TensorFlow Lite:

   yolo export model=best.pt format=tflite  # For TensorFlow Lite

2. Integrate the exported model into your STM32 project. TensorFlow Lite provides support for microcontrollers, which can be really handy for deploying models onto embedded systems like the STM32. You might find TensorFlow Lite for Microcontrollers' documentation particularly useful for this.

Remember, the key to a smooth deployment process is ensuring your model is optimized and compatible with your target environment. With your model exported to TensorFlow or TensorFlow Lite, you'll be well on your way to incorporating AI capabilities into your STM32 projects with a RealSense 3D camera. 🚀

Keep up the great work, and don't hesitate to reach out if you need further assistance!

[Noirewinter]

Why wasn't NMS added to the tflite export? I believe this is a very common requirement.

[glenn-jocher]

@usama1123456789 hey 👋! It looks like the issue you're encountering is due to the spaces and parentheses in your model file's name. In shell commands, spaces and special characters can lead to syntax errors or unintended behavior. You can resolve this by enclosing the path in quotes and escaping the parentheses:

!yolo export model='/content/yolov8nsss \(3\).pt' format=tflite

Alternatively, you could rename your model file to eliminate spaces and parentheses, which simplifies command usage without needing to escape characters. For example:

# Rename the model file
mv '/content/yolov8nsss (3).pt' /content/yolov8nsss_3.pt

# Then run the export command
!yolo export model=/content/yolov8nsss_3.pt format=tflite

Hope this helps you get past the error and successfully convert your YOLOv8n model to TFLite format! 🚀

[Noirewinter]

@glenn-jocher Thank you~But this method would require importing the TensorFlow library into the mobile app, which would significantly increase the size of the package, right?
So I guess if we don't import it, we'll just have to manually implement an NMS to handle it.

[glenn-jocher]

@Noirewinter hey there! 🌟 You're absolutely right. Incorporating the TensorFlow library just to apply NMS might indeed bump up the package size for mobile applications, which isn't ideal given storage and performance considerations on those platforms.

Going the route of manually implementing NMS can be a thoughtful solution, ensuring you keep your app lean while still effectively filtering detections. For most mobile deployment scenarios, a simple NMS logic can be pretty straightforward to code up and optimise for speed.

Just in case it helps, here's a quick pseudo-code for a basic NMS implementation you could adapt:

def nms(boxes, scores, iou_threshold):
    # Sort scores and their corresponding boxes
    sorted_indices = np.argsort(scores)[::-1]
    keep = []
    while sorted_indices.size > 0:
        # Pick box with highest score and add its index to keep list
        current_box_index = sorted_indices[0]
        keep.append(current_box_index)
        if sorted_indices.size == 1:
            break
        # Compute IoU of this box with all others
        current_box = boxes[current_box_index]
        other_boxes = boxes[sorted_indices[1:]]
        ious = compute_iou(current_box, other_boxes)
        # Only keep boxes with IoU less than threshold
        sorted_indices = sorted_indices[1:][ious < iou_threshold]
    return keep

Remember, the efficient implementation of compute_iou depends considerably on your application requirements and platform constraints.

Happy coding, and we're here if you have more questions! 🚀

[Noirewinter]

Thanks, I'll try it. I also found cv2.dnn.NMSBoxes to be a nice implementation of NMS, since we had already imported cv2 library into the project for image processing.

[glenn-jocher]

Hey there! 👋 Yes, incorporating cv2.dnn.NMSBoxes as your NMS solution is a great approach, especially if you're already leveraging the OpenCV library for image processing tasks. This not only optimizes the post-processing stage by reusing existing resources but also keeps your application package lean. Plus, OpenCV is highly optimized for performance, so it should work smoothly for your needs.

If you need a quick refresher or code snippet on using cv2.dnn.NMSBoxes, here it is:

indices = cv2.dnn.NMSBoxes(boxes, scores, score_threshold, nms_threshold)

• boxes should contain your detected bounding boxes.
• scores are the confidence scores for each box.
• score_threshold is the minimum score for a box to be considered.
• nms_threshold is the IOU threshold for considering two boxes as one.

Keep rocking with your project, and feel free to reach out if you have more questions or need further assistance! 🚀

[Aya-Annabi]

i need to make make infrence with best.pt on jetson nano board so after i exported it to .engine format in my pc and i transfered it to jetson nano got error says exported with different version that 8.0.1.6 how can I export it to tensorrt version=8.0.1.6 note that my tensorrt version in my pc is 10.0.0b6

[pderrenger]

It looks like you’re experiencing issues because of the TensorRT version mismatch between your PC and the Jetson Nano. To resolve this, you can try exporting your model directly on the Jetson Nano where TensorRT 8.0.1.6 is installed. This ensures that the exported .engine file is compatible with your deployment environment.

Here’s how you can do it:

1. Transfer your best.pt model file to the Jetson Nano.
2. Set up your environment on the Jetson Nano to run Ultralytics YOLOv8, ensuring TensorRT 8.0.1.6 and other dependencies are correctly installed.
3. Use the following command to export your model on the Jetson Nano:

yolo export model=best.pt format=engine

Make sure to activate the correct Python environment if you’re using a virtual environment. This approach should eliminate the version conflict. If you encounter any further issues, feel free to ask for more help! 🚀

[Piebee007]

Hi,
2 questions:
1 - exporting to any tensorflow based format doesnt work and i keep getting a "KerasTensor cannot be used as input to a Tensorflow function error"
2 - I have exported to onnx and imported into Unity. The output shape is (1,1,8400,6). How can i interpret this data for object detection with 2 classes? I followed this tutorial but the output shape is completelty different and don't know what to dohttps://www.youtube.com/watch?v=ggmArUbRvC4

[glenn-jocher]

Hello!

1. TensorFlow Export Issue: It sounds like you're running into issues related to TensorFlow's handling of KerasTensor inputs. Ensure that you're using the latest versions of TensorFlow and keras. Sometimes, wrapping TensorFlow operations in Lambda layers helps to ensure compatibility. Here's an example you might find helpful:

   from tensorflow.keras.layers import Lambda
   from tensorflow.keras import backend as K
   
   model.add(Lambda(lambda x: K.tf_function(x)))

2. Interpreting ONNX Output in Unity: The output shape (1, 1, 8400, 6) likely represents [batch, num_boxes, coords+score+classes]. Each box is defined by its coordinates, a confidence score and class probabilities, which adds up to 6 values:

   • x_center, y_center: The center coordinates.
   • width, height: Box dimensions.
   • confidence: The likelihood of a box containing an object.
   • class_probabilities: Probability distribution over the two classes.

In Unity, you'd loop over these bounding boxes and utilize the coordinates and class probabilities to assign and render detections.

For further refinements or specific technical questions, don't hesitate to consult the Ultralytics export documentation.

Hope this helps!

[Hisham-Imran-Ssengendo]

Greetings members. I have tried changing my trained pytorch model to tensorflow lite model but am getting many errors, some of them relating to parameters used.
Please help me out.
I want to change my custom trained pytorch model to tensorflow lite and make object detection using bounding boxes and my class's index.

[glenn-jocher]

Hello! It looks like you're having some issues exporting your custom-trained PyTorch model to TensorFlow Lite for object detection. Here’s a simplified step you can follow to perform this conversion:

1. Export your PyTorch model to ONNX format — Start by exporting your PyTorch model to ONNX, which acts as an intermediary format:

   from ultralytics import YOLO
   
   # Load your custom-trained model
   model = YOLO('path/to/your/custom_model.pt')
   
   # Export to ONNX
   model.export(format='onnx', imgsz=(640, 640))  # adjust 'imgsz' as needed

2. Convert ONNX model to TensorFlow Lite — Next, use tools like ONNX TensorFlow to convert the ONNX model to TensorFlow, and then TensorFlow Lite Converter to convert the TensorFlow model to TFLite. Unfortunately, direct conversion tools from ONNX to TFLite might not handle all operations supported in YOLO models directly, so intermediate conversion to TensorFlow is often required.

For details on arguments for export and further optimizations like quantization which might help you in reducing the model size while speeding up the inference especially on edge devices, please refer to our Export documentation.

Remember, getting the conversion right might require tweaking certain parameters based on specific layers and operations used in your model. If you still face any issues, could you please share the specific errors you're encountering? That would help in providing more targeted assistance! 😊👍

[Hisham-Imran-Ssengendo]

Okay then. Thank you very much. Let me first try this. Kind regards Hisham Imran Ssengendo.

…

On Tue, 14 May 2024 at 00:21, Glenn Jocher ***@***.***> wrote: Hello! It looks like you're having some issues exporting your custom-trained PyTorch model to TensorFlow Lite for object detection. Here’s a simplified step you can follow to perform this conversion: 1. *Export your PyTorch model to ONNX format* — Start by exporting your PyTorch model to ONNX, which acts as an intermediary format: from ultralytics import YOLO # Load your custom-trained modelmodel = YOLO('path/to/your/custom_model.pt') # Export to ONNXmodel.export(format='onnx', imgsz=(640, 640)) # adjust 'imgsz' as needed 2. *Convert ONNX model to TensorFlow Lite* — Next, use tools like ONNX TensorFlow <https://github.com/onnx/onnx-tensorflow> to convert the ONNX model to TensorFlow, and then TensorFlow Lite Converter <https://www.tensorflow.org/lite/convert> to convert the TensorFlow model to TFLite. Unfortunately, direct conversion tools from ONNX to TFLite might not handle all operations supported in YOLO models directly, so intermediate conversion to TensorFlow is often required. For details on arguments for export and further optimizations like quantization which might help you in reducing the model size while speeding up the inference especially on edge devices, please refer to our Export documentation <https://docs.ultralytics.com/modes/export/>. Remember, getting the conversion right might require tweaking certain parameters based on specific layers and operations used in your model. If you still face any issues, could you please share the specific errors you're encountering? That would help in providing more targeted assistance! 😊👍 — Reply to this email directly, view it on GitHub <#7933 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWWZRMYY4E2KAOWRAU7P4KDZCEVFPAVCNFSM6AAAAABCTE3SQOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMRWGE2TA> . You are receiving this because you commented.Message ID: ***@***.*** com>

[pderrenger]

Hello Hisham,

Great to hear that you're giving it a try! If you encounter any issues or have further questions during the process, feel free to reach out. We're here to help! 😊

[cfywg]

Hello, I am a rookie. I tried to replace the YOLOv8 backbone with the convnextv2 network and tried to convert the modified generated PT model file into a tflite format model file through export.py. However, only onnx could be generated, and an error occurred when generating tflite.
How can I use export.py to convert the modified generated pt model into a tflite model ？

[pderrenger]

@cfywg hello! Great step experimenting with the YOLOv8 backbone and ConvNeXtV2 integration. 🚀 If you're encountering issues exporting to TFLite, it's likely a compatibility issue between the PT model's layers or operations and what TFLite currently supports.

Here's a general example of how to export a model to TFLite using the command line, but keep in mind that every model's compatibility may vary depending on the operations used:

yolo export model=path/to/your/custom_model.pt format=tflite

If this process results in errors, I recommend checking:

1. If your model uses operations not supported by TFLite.
2. If any custom layers or modifications need specific adjustments or implementations in TFLite.

Make sure to consult TensorFlow's TFLite documentation for specifics about operation support and any potential workarounds. Sometimes, small tweaks to the model architecture can solve these issues.

If you continue to have issues, please feel free to provide the error message and details about your model's architecture for more targeted advice! 🛠️

[GuenKainto]

in onnx if i set simplify = true, how it works, whether it reduces any parameters such as mAP, Presision, Recall of the model

[glenn-jocher]

When you set simplify=True during the ONNX export of your YOLOv8 model, it simplifies the model graph by removing redundant nodes and merging operations where possible. This process is designed to optimize the model for better performance and compatibility, especially with different ONNX runtime environments.

Typically, simplifying the model should not reduce its accuracy metrics like mAP, Precision, or Recall as long as the simplifications do not alter the computational logic of the model. However, it's always a good practice to validate the performance of the simplified model to ensure that there are no unexpected changes in its behavior or accuracy.

Here's a quick example of how you might export a model with simplification:

from ultralytics import YOLO

# Load a model
model = YOLO('yolov8n.pt')

# Export the model with simplification
model.export(format='onnx', simplify=True)

If you encounter any issues or notice a drop in performance, it might be helpful to compare the outputs of the simplified model against the original to identify any discrepancies.

[shimaamorsy]

When I tried to run yolov8-seg.onnx with the batching option activated, this error appeared ..

ort-wasm-simd.jsep.js:54 2024-05-17 15:23:15.490199 [W:onnxruntime:, session_state.cc:1162 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf

ort-wasm-simd.jsep.js:54 2024-05-17 15:23:15.492300 [W:onnxruntime:, session_state.cc:1164 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

ERROR_MESSAGE: Non-zero status code returned while running Softmax node. Name:'/model.22/dfl/Softmax' Status Message: Failed to run JSEP kernel

To reproduce

To Produce

from ultralytics import YOLO

# Load a model
model = YOLO('yolov8n.pt')  # load an official model
model = YOLO('path/to/best.pt')  # load a custom trained model

# Export the model
model.export(format='onnx' ,dynmic=True)

Here'is the mdel

Script

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no"/>
    <title>Yolov8 test</title>

    <!-- ONNX RUNTIME -->
    <script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.webgpu.min.js"></script>
  </head>
  <body>
   <script>
    const modelName = "yolov8n-seg-batching.onnx";
    const modelInputShape = [1, 3, 640, 640];
    async function testModel() {
    
        let model =  await ort.InferenceSession.create(modelName,{ executionProviders: ["webgpu","cpu"] });
        const tensor = new ort.Tensor("float32",new Float32Array(modelInputShape.reduce((a, b) => a * b)),modelInputShape);
        await model.run({ images: tensor });
        console.log(model);
    }
    testModel();
    </script>
  </body>
</html>

Urgency

It's somewhat urgent

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.16.3

Execution Provider

'webgpu' (WebGPU)

[pderrenger]

It looks like you encountered some issues when trying to run your YOLOv8-seg model in ONNX format with batching enabled, specifically regarding the Softmax node.

The warning about nodes not being assigned to preferred execution providers often isn't critical but can affect performance if certain operations aren't optimized for the chosen backend (WebGPU in your case).

Regarding the error with the Softmax node, this could potentially be related to incompatibilities or unsupported operations in the ONNX model when used with the ONNXruntime's WebGPU execution provider.

A possible workaround could be to try different execution providers, such as WebAssembly (WASM), if performance is less of a concern compared to running the model successfully. Adjusting this in your HTML script would look like:

let model = await ort.InferenceSession.create(modelName, { executionProviders: ["wasm"]});

Also, make sure that your ONNX model is exported with settings that are compatible with your target runtime environment. Here's a slightly modified export line that might help improve compatibility:

model.export(format='onnx', dynamic=True, simplify=True, opset=12)

These changes ensure the model is using a potentially more compatible opset version and simplifies the model graph which might resolve issues related to unsupported operations in specific execution providers.

If the issue persists, consider raising an issue in the ONNXruntime repository, as the community there might have more targeted insights into handling such specific errors in the WebGPU backend. 🚀

[shimaamorsy]

Hi @pderrenger , @glenn-jocher

I exported my model using the following code:

model.export(format='onnx', dynamic=True, simplify=True, opset=12)

However, the exported ONNX model currently runs on the GPU only when the batch size is 1.
I need to run the model on the GPU with a batch size greater than 1.

Can you assist me with this task? Thank you in advance!

[pderrenger]

Hi @shimaamorsy,

Thanks for reaching out! To run your ONNX model on the GPU with a batch size greater than 1, ensure your export settings are correct. You can try the following:

model.export(format='onnx', dynamic=True, simplify=True, opset=12, batch=4)

This sets the batch size to 4. Adjust the batch parameter as needed. If issues persist, consider using a different execution provider or checking the ONNX model compatibility with your GPU.

For more details, visit the Ultralytics Export Guide.

Best of luck! 🚀

[shimaamorsy]

@pderrenger
@glenn-jocher
Unfortunately, the same problem persists.

[pderrenger]

Hi @shimaamorsy,

Thank you for your patience. It seems the issue persists despite the adjustments. Let's try a couple more steps to resolve this:

1. Check ONNX Model Compatibility: Ensure the exported ONNX model is fully compatible with the WebGPU execution provider. Sometimes, certain operations may not be supported or optimized for WebGPU.

2. Experiment with Different Opset Versions: While opset 12 is generally stable, you might want to try exporting with a higher opset version, such as 13 or 14, to see if it resolves the issue:

   model.export(format='onnx', dynamic=True, simplify=True, opset=13)

3. Use WebAssembly (WASM) Execution Provider: If WebGPU continues to cause issues, consider switching to the WASM execution provider, which might offer better compatibility:

   let model = await ort.InferenceSession.create(modelName, { executionProviders: ["wasm"] });

If these steps don't resolve the issue, please provide more details or logs, and we can further investigate. Thank you for your understanding and cooperation! 😊