guides/coral-edge-tpu-on-raspberry-pi/

[giscus[bot]]

guides/coral-edge-tpu-on-raspberry-pi/

Guide on how to use Ultralytics with a Coral Edge TPU on a Raspberry Pi for increased inference performance.

https://docs.ultralytics.com/guides/coral-edge-tpu-on-raspberry-pi/

[DAVIDNYARKO123]

Exactly the file I was looking for to reignite the Coral TPU dream.

[glenn-jocher]

Glad to hear the guide is what you were looking for! 🌟 If you're diving into the Coral TPU with a Raspberry Pi, just follow the steps outlined in the guide to get everything up and running. If you hit any snags or have questions, feel free to drop them here. Happy inferencing!

[ukicomputers]

If I understood right, this code only converts YOLO model for edgetpu platform:

from ultralytics import YOLO

# Load a model
model = YOLO('path/to/model.pt')  # Load a official model or custom model

# Export the model
model.export(format='edgetpu')

But after, it says to run the model with YOLO, with following code:

from ultralytics import YOLO

# Load a model
model = YOLO('path/to/edgetpu_model.tflite')  # Load a official model or custom model

# Run Prediction
model.predict("path/to/source.png")

So then we are not running the model on Coral, but on our local device, right? If that is true, please say if we need to use that exported model with something like pycoral or libcoral. Thanks in advance.

[ukicomputers]

Thanks @glenn-jocher! I converted the default YOLOv8 model using following Collab notebook: here.
I used your code for detection (btw it is required to change Image.ANTIALIAS to Image.LANCZOS, because it's deprecated), and when running it, the Edge TPU tries to load a model, but it responds with:

F driver/usb/usb_driver.cc:857] transfer on tag 1 failed. Abort. Deadline exceeded: USB transfer error 2 [LibUsbDataOutCallback]

As I researched the internet, it seems like Edge TPU DOES NOT SUPPORT 80 classes with high resolutions (I was using precompiled yolov8n.pt). The recommended (probably required settings) are:

• 640x640 train image resolution
• Trained 2 classes (this is not probably correct, I HAVE NOT yet tested anything)
  The further testing is required, and if conversion to edgetpu means just to Coral (it is the only "Edge TPU" device, if I am correct), then the some PR should fix the issues (but it does not get in context in YOLO library, so what do you say?).

I will be updated.

[glenn-jocher]

@ukicomputers hey there! Thanks for diving into this and sharing your findings and that Colab notebook. 📔 It's super helpful for the community!

Indeed, you've run into an interesting challenge. When exporting models for the Edge TPU, like Coral, it's essential to match the TPU's hardware capabilities. This often means adapting the model and/or input sizes, and the number of classes could be a factor as well due to the computational complexity and memory constraints of the TPU.

Running the exported .tflite model with pycoral or libcoral is absolutely the right approach for leveraging the Coral Edge TPU acceleration. The model.predict() code snippet indeed runs the model locally, not on the TPU. For Edge TPU acceleration, code like this should come in handy:

from pycoral.adapters import detect
from pycoral.adapters import common
from pycoral.utils.edgetpu import make_interpreter
from PIL import Image

interpreter = make_interpreter('path/to/edgetpu_model.tflite')
interpreter.allocate_tensors()

image = Image.open('path/to/source.png')
_, scale = common.set_resized_input(interpreter, image.size, lambda size: image.resize(size, Image.LANCZOS))
interpreter.invoke()

detections = detect.get_objects(interpreter, scale)
# Handle detections here

Regarding the two-class model and resolution considerations, testing different configurations to identify the optimal setup for your specific use case is indeed a good plan. It's a bit of a balancing act between model complexity and TPU's computational resources.

Let's keep each other updated on further developments or discoveries! And yup, contributions or fixes via PRs are always welcome if you stumble upon something revolutionary! 😄 Keep on exploring!

[aeltersCDS]

@glenn-jocher is there any way to run the Object Counting solution in the edge tpu?

[Aishneet]

Hey @glenn-jocher,
I am confused regarding the inference behavior of models, particularly those utilizing the EdgeTPU, within the Ultralytics library. Just above you mentioned that when using model.predict() with an EdgeTPU model, the inference runs locally rather than on the TPU. However, upon inspecting the code in the autobackend module on GitHub, it seems that when the model is an EdgeTPU model, delegates are loaded, tensors are allocated, and inference is executed on the TPU.

This has left me confused about the actual behavior. Could you kindly provide clarification on whether EdgeTPU models run inference locally or on the TPU?

[glenn-jocher]

Hey there! 😊 It looks like there might be a bit of confusion regarding the inference behavior with Edge TPU models within our library. Let's clear that up!

When you run model.predict() with an Edge TPU model in our Ultralytics library, the inference indeed aims to utilize the Edge TPU for acceleration. So, if the setup is correct (Coral device connected and necessary libraries installed on your system), the inference should run on the Edge TPU, not just locally on the CPU.

The details you've pointed out concerning delegates and tensor allocations in our autobackend module are crucial. They're part of the mechanism that ensures the model utilizes the TPU when available. It selects the Edge TPU delegate for TensorFlow Lite models, which directs the inference operations to run on the Coral Edge TPU.

However, to ensure everything runs smoothly, it's vital to have the correct version of the Coral Edge TPU runtime installed and to follow the setup guide carefully. Misconfigurations or mismatched library versions could potentially lead to unexpected behaviors or errors.

If you're encountering issues, double-checking the Edge TPU runtime installation and ensuring your Raspberry Pi setup matches the requirements might help. Also, using pycoral or libcoral for inference is an excellent approach to leverage the TPU further effectively.

Your observation and research definitely push us towards providing clearer documentation and support for users keen on leveraging Edge TPU acceleration. Thanks a ton for bringing this to our attention! Let me know if there's anything else I can help with. 🚀

[opc241]

Hello,

i would also add a question here, since i am running a yolov8 model on a Raspberry Pi 4, baught a Coral TPU Accelerator, installed everything like described, but the following code runs into the following issue. Since there is no real pycoral support anymore, i try my luck here, maybe someone has an idea how to fix that.

from pycoral.adapters import detect
from pycoral.adapters import common
from pycoral.utils.edgetpu import make_interpreter
from PIL import Image

interpreter = make_interpreter('/home/pi/homecontrol/software/yolo-Weights/yolov8n_full_integer_quant.tflite')
interpreter.allocate_tensors()

image = Image.open('/home/pi/homecontrol/software/yolo-Weights/yolov8n_saved_model/capture.png')
_, scale = common.set_resized_input(interpreter, image.size, lambda size: image.resize(size, Image.LANCZOS))

# Run inference
interpreter.invoke()

# Process detection results
detections = detect.get_objects(interpreter, scale)
for detection in detections:
    print(detection)

And the output is:

Traceback (most recent call last):
  File "/home/pi/homecontrol/software/__test.py", line 16, in <module>
    detections = detect.get_objects(interpreter, scale)
  File "/usr/lib/python3/dist-packages/pycoral/adapters/detect.py", line 210, in get_objects
    count = int(interpreter.tensor(signature['outputs']['output_0'])()[0])
KeyError: 'output_0'

Greetings Markus

[glenn-jocher]

@opc241 hi Markus,

It seems like the issue here is with accessing the output tensor using a key that might not exist in your model's output dictionary. A quick solution could be to try using an integer index to access the first output tensor, as your model might have a different output name than 'output_0'. This can vary based on how the model was converted to TFLite. Here's how you could modify the line throwing the KeyError:

# Assuming your model has a single primary output, you can try accessing it with index [0]
count = int(interpreter.get_tensor(interpreter.get_output_details()[0]['index'])[0])

This line replaces the one causing the 'output_0' KeyError, and it dynamically fetches the first output tensor regardless of its name.

Make sure your Edge TPU is using the correct runtime version and is compatible with your TensorFlow Lite version, as mentioned in the Coral Edge TPU guide on our docs for Raspberry Pi setups. These steps are crucial, especially since projects like Coral can have specific compatibility requirements.

Let me know if this helps or if you see any other issues! 😊

[opc241]

Hi Glenn,

and thanks a lot for your reply. I implemented your suggested fix and this leads to the following:

Traceback (most recent call last):
  File "/home/pi/homecontrol/software/__test.py", line 16, in <module>
    detections = detect.get_objects(interpreter, scale)
  File "/usr/lib/python3/dist-packages/pycoral/adapters/detect.py", line 211, in get_objects
    count = int(interpreter.get_tensor(interpreter.get_output_details()[0]['index'])[0])
TypeError: only size-1 arrays can be converted to Python scalars

Good hint with the versions. I often read about that and the currently installed versions are:
Name: tflite-runtime
Version: 2.13.0

Name: pycoral
Version: 2.0.0

Python 3.9.2

Raspberry Pi 4: Debian GNU/Linux 11 (bullseye)

And the way i exported the yolov8 to .tflite, was with this code snippet:
(after export i chose the yolov8n_full_integer_quant.tflite model)

model = YOLO("yolo-Weights/yolov8n.pt")
model.export(format="edgetpu", imgsz=640)

Is this the proper way of exporting a yolov8 --> .tflite?

I am thankful for every hint, which i will follow.

[glenn-jocher]

Hi there!

Glad to hear you've implemented the fix, though it looks like you've hit another snag. The error you're encountering seems to point to an issue with how the model's output is being interpreted. It might be due to expecting a scalar output where there's actually an array.

Regarding your model export process to .tflite for the Coral Edge TPU, your steps look correct. However, for utilizing the Edge TPU, it's essential to ensure the model is accurately quantized and compatible. The method you've used should give you a quantized .tflite model suitable for the Edge TPU as long as the model architecture supports it. For best results, consider ensuring your model's architecture is fully compatible with Edge TPU requirements.

Here's a handy checklist:

• Verify model requirements for Edge TPU compatibility. Complex operations may not be supported.
• Make sure you're using the correct version of the Edge TPU compiler and runtime that match your hardware and software versions.
• Run the model conversion and compilation steps on a non-ARM platform like Google Colab or an x86_64 machine as recommended in our docs.

For example, to convert and check compatibility, you can do:

from ultralytics import YOLO

model = YOLO('yolo-Weights/yolov8n.pt')
model.export(format='edgetpu', imgsz=640)

And then use the appropriate Edge TPU compiler to ensure compatibility.

I hope these suggestions help. Let me know if you have any other questions or run into more issues! 🚀

[opc241]

Hi Glenn,

Of course i tried your suggestions, so i used Google Colab to compile the .tflite model to a "edgetpu.tflite" model, but now the following error occurs by running the (not changed) code:

Traceback (most recent call last):
  File "/home/pi/homecontrol/software/__test.py", line 17, in <module>
    detections = detect.get_objects(interpreter, scale)
  File "/usr/lib/python3/dist-packages/pycoral/adapters/detect.py", line 214, in get_objects
    elif common.output_tensor(interpreter, 3).size == 1:
  File "/usr/lib/python3/dist-packages/pycoral/adapters/common.py", line 29, in output_tensor
    return interpreter.tensor(interpreter.get_output_details()[i]['index'])()
IndexError: list index out of range

If you have another hint for that i will try, but to be honest i don't feel fun with Google's Coral TPU Accelerator.
This topic is not intuitive/compatible (enough) for me - with focus on all of the dependencies and the need of "older" versions of python [...]

I had a simple code, using YOLOv8 and the predict() method, but this unfortunately uses too much CPU on the Raspberry Pi. That is why i thought Google Coral could be it, but hm ... maybe not.

Thanks for your help and if you have a clue why this error now occurs, feel free to tell me, otherwise i think i will send the Coral back to Amazon :-(

[pderrenger]

Hey there,

I'm truly sorry to hear about the stumbling blocks you've encountered with the Coral TPU setup. It's indeed quite a leap from running straightforward inference with YOLOv8 on CPU, and I understand how the complexity with dependencies can be quite daunting.

The error you're facing suggests an issue with accessing the output tensor indices. This could be stemming from a mismatch between the expected output format by the Edgetpu model and the actual format being generated. An initial step could be to double-check if the model converted to edgetpu.tflite is fully compatible and correctly quantized.

Here's a small something you might want to try - ensuring the TFLite runtime and Edge TPU software are correctly set up and aligned with your model's requirements:

# First, ensure the tflite-runtime is using the correct version.
pip install --upgrade tflite-runtime

# If using the CLI export command, also ensure the model format matches the expected Edge TPU architecture.
yolo export model=path/to/model.pt format=edgetpu

If the quirk persists, it might be rooted in the conversion process itself or how the inference script is interpreting the output tensors. If you're willing to share a bit more about your model conversion code or the script leading to the error, I'd be keen to dive deeper.

Usage of Coral with Raspberry Pi indeed brings a notable inference speed boost, but setup woes can sometimes overshadow the benefits. Don't hesitate to reach back if there's more to dissect or if a fresh set of logs might offer new clues. Meanwhile, holding off on returning the Coral might still be a good call - let's troubleshoot a bit more! 🛠️

[StefansAI]

I followed the instructions above but still have problems getting it to work. After installing Ultralytics in an environment I run following:

wget https://github.com/feranick/libedgetpu/releases/download/16.0TF2.16.1-1/libedgetpu1-std_16.0tf2.16.1-1.bookworm_arm64.deb
sudo dpkg -i libedgetpu1-std_16.0tf2.16.1-1.bookworm_arm64.deb

wget https://github.com/feranick/libedgetpu/releases/download/16.0TF2.16.1-1/libedgetpu-dev_16.0tf2.16.1-1.bookworm_arm64.deb
sudo dpkg -i libedgetpu-dev_16.0tf2.16.1-1.bookworm_arm64.deb

#https://github.com/JungLearnBot/RPi5_yolov8/blob/main/Readme.RPi5.coral_tpu.picam.qt.md
wget https://github.com/oberluz/pycoral/releases/download/2.13.0/pycoral-2.13.0-cp311-cp311-linux_aarch64.whl
pip install pycoral-2.13.0-cp311-cp311-linux_aarch64.whl --no-deps
pip install tflite-runtime==2.14.0

export always fails:

Ultralytics YOLOv8.1.43 🚀 Python-3.11.2 torch-2.2.2 CPU (Cortex-A76)
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs

�[34m�[1mPyTorch:�[0m starting from 'yolov8n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (6.2 MB)

�[34m�[1mTensorFlow SavedModel:�[0m starting export with tensorflow 2.16.1...
WARNING ⚠️ tensorflow<=2.13.1 is required, but tensorflow==2.16.1 is currently installed https://github.com/ultralytics/ultralytics/issues/5161

�[34m�[1mONNX:�[0m starting export with onnx 1.16.0 opset 17...
�[34m�[1mONNX:�[0m simplifying with onnxsim 0.4.36...
�[34m�[1mONNX:�[0m export success ✅ 2.6s, saved as 'yolov8n.onnx' (12.3 MB)
�[34m�[1mTensorFlow SavedModel:�[0m starting TFLite export with onnx2tf 1.17.5...

�[07mAutomatic generation of each OP name started�[0m ========================================
�[32mAutomatic generation of each OP name complete!�[0m

�[07mModel loaded�[0m ========================================================================

�[07mModel conversion started�[0m ============================================================
�[33mWARNING:�[0m The optimization process for shape estimation is skipped because it contains OPs that cannot be inferred by the standard onnxruntime.
�[33mWARNING:�[0m module 'onnx' has no attribute '_serialize'
�[31mERROR:�[0m The trace log is below.
Traceback (most recent call last):
  File "/home/pi/yolo_env/lib/python3.11/site-packages/onnx2tf/utils/common_functions.py", line 288, in print_wrapper_func
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/yolo_env/lib/python3.11/site-packages/onnx2tf/utils/common_functions.py", line 361, in inverted_operation_enable_disable_wrapper_func
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/yolo_env/lib/python3.11/site-packages/onnx2tf/ops/Conv.py", line 246, in make_node
    input_tensor = get_padding_as_op(
                   ^^^^^^^^^^^^^^^^^^
  File "/home/pi/yolo_env/lib/python3.11/site-packages/onnx2tf/utils/common_functions.py", line 2009, in get_padding_as_op
    return tf.pad(x, padding)
           ^^^^^^^^^^^^^^^^^^
  File "/home/pi/yolo_env/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/pi/yolo_env/lib/python3.11/site-packages/keras/src/backend/common/keras_tensor.py", line 91, in __tf_tensor__
    raise ValueError(
ValueError: A KerasTensor cannot be used as input to a TensorFlow function. A KerasTensor is a symbolic placeholder for a shape and dtype, used when constructing Keras Functional models or Keras Functions. You can only use it as input to a Keras layer or a Keras operation (from the namespaces `keras.layers` and `keras.operations`). You are likely doing something like:

x = Input(...)
...
tf_fn(x) # Invalid.

What you should do instead is wrap `tf_fn` in a layer:

class MyLayer(Layer):
def call(self, x):
return tf_fn(x)

x = MyLayer()(x)

�[31mERROR:�[0m input_onnx_file_path: yolov8n.onnx
�[31mERROR:�[0m onnx_op_name: wa/model.0/conv/Conv
�[31mERROR:�[0m Read this and deal with it. https://github.com/PINTO0309/onnx2tf#parameter-replacement
�[31mERROR:�[0m Alternatively, if the input OP has a dynamic dimension, use the -b or -ois option to rewrite it to a static shape and try again.
�[31mERROR:�[0m If the input OP of ONNX before conversion is NHWC or an irregular channel arrangement other than NCHW, use the -kt or -kat option.
�[31mERROR:�[0m Also, for models that include NonMaxSuppression in the post-processing, try the -onwdt option.

I ran the export in colab and the export worked, even though it errors out there too. So I used the colab exports and copied them to the RPi5, created the python file with make_interpreter, but get this error as described here too:

Traceback (most recent call last):
  File "/home/pi/tpu_detect.py", line 13, in <module>
    detections = detect.get_objects(interpreter, scale)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/yolo_env/lib/python3.11/site-packages/pycoral/adapters/detect.py", line 210, in get_objects
    count = int(interpreter.tensor(signature['outputs']['output_0'])()[0])
                                   ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'output_0'

What am I missing? Has anyone applied this to an RPi5 successfully?

[pderrenger]

Hey there! 🌟 It seems like you're encountering some issues with exporting your model and running it successfully on the Raspberry Pi. Let's see if we can address this.

Firstly, regarding the TensorFlow version warning during export, the recommended TensorFlow version for optimal compatibility is <=2.13.1. However, since you managed to export successfully in Colab, we'll focus on the next steps.

Concerning the error you're facing when trying to run the model with the Edge TPU on your RPi5, it looks like there might be an issue with how the model outputs or Tensor names are expected by the pycoral library. The KeyError: 'output_0' suggests that the expected output tensor name doesn't match.

One potential solution is to explicitly define the output tensor names when exporting your model. However, given the complexity and potential changes in both the exporting process and how pycoral accesses model outputs, this might not always be straightforward.

Since the Coral project has had limited updates, and depending on the specific versions of TensorFlow Lite and pycoral you're working with, compatibility issues might arise. If you haven't already, make sure you're using the versions mentioned in the updated guide for Raspberry Pi with Coral Edge TPU. This might help mitigate some of the version mismatch issues.

As a temporary workaround, double-check the versions of all libraries and runtimes involved. It may also help to explore any available forums or GitHub issues related to pycoral and TensorFlow Lite compatibility for further insights.

Sorry to hear about the trouble this is causing you! If these steps don't resolve your issue, please consider providing the model or more details about the error, and I'd be happy to take a closer look. Hope this helps a bit! 🚀

[StefansAI]

@pderrenger, thanks a lot for the quick reply.
I had TensorFlow 2.16.1 installed, it may have been installed by the export function as dependency. After downgrading to 2.13.1 I could run the export successfully on the Rpi5. It still had an error at the end, not being able to install the edgetpu-compiler, which is not available anymore.

Ultralytics YOLOv8.1.43 🚀 Python-3.11.2 torch-2.2.2 CPU (Cortex-A76)
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs

�[34m�[1mPyTorch:�[0m starting from 'yolov8n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (6.2 MB)

�[34m�[1mTensorFlow SavedModel:�[0m starting export with tensorflow 2.13.1...

�[34m�[1mONNX:�[0m starting export with onnx 1.16.0 opset 17...
�[34m�[1mONNX:�[0m simplifying with onnxsim 0.4.36...
�[34m�[1mONNX:�[0m export success ✅ 2.7s, saved as 'yolov8n.onnx' (12.3 MB)
�[34m�[1mTensorFlow SavedModel:�[0m starting TFLite export with onnx2tf 1.17.5...

�[07mAutomatic generation of each OP name started�[0m ========================================
�[32mAutomatic generation of each OP name complete!�[0m

�[07mModel loaded�[0m ========================================================================

�[07mModel conversion started�[0m ============================================================
�[33mWARNING:�[0m The optimization process for shape estimation is skipped because it contains OPs that cannot be inferred by the standard onnxruntime.
�[33mWARNING:�[0m module 'onnx' has no attribute '_serialize'
�[07msaved_model output started�[0m ==========================================================
�[32msaved_model output complete!�[0m
�[32mFloat32 tflite output complete!�[0m
�[32mFloat16 tflite output complete!�[0m
�[34mInput signature information for quantization�[0m
�[34msignature_name�[0m: serving_default
�[34minput_name.0�[0m: images �[34mshape�[0m: (1, 640, 640, 3) �[34mdtype�[0m: <dtype: 'float32'>
�[32mDynamic Range Quantization tflite output complete!�[0m
�[32mINT8 Quantization tflite output complete!�[0m
�[32mFull INT8 Quantization tflite output complete!�[0m
�[32mINT8 Quantization with int16 activations tflite output complete!�[0m
�[32mFull INT8 Quantization with int16 activations tflite output complete!�[0m
�[34m�[1mTensorFlow SavedModel:�[0m export success ✅ 291.0s, saved as 'yolov8n_saved_model' (41.0 MB)
�[34m�[1mEdge TPU:�[0m WARNING ⚠️ Edge TPU known bug https://github.com/ultralytics/ultralytics/issues/1185

�[34m�[1mEdge TPU:�[0m export requires Edge TPU compiler. Attempting install from https://coral.ai/docs/edgetpu/compiler/
OK
deb https://packages.cloud.google.com/apt coral-edgetpu-stable main
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian-security bookworm-security InRelease
Hit:3 http://deb.debian.org/debian bookworm-updates InRelease
Hit:4 https://packages.cloud.google.com/apt coral-edgetpu-stable InRelease
Hit:5 http://archive.raspberrypi.com/debian bookworm InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
�[34m�[1mEdge TPU:�[0m export failure ❌ 3.4s: Command 'sudo apt-get install edgetpu-compiler' returned non-zero exit status 100.

I downloaded the bus image and ran the suggested python code:

from pycoral.adapters import detect
from pycoral.adapters import common
from pycoral.utils.edgetpu import make_interpreter
from PIL import Image

interpreter = make_interpreter('yolov8n_saved_model/yolov8n_full_integer_quant.tflite')
interpreter.allocate_tensors()

image = Image.open('bus.jpg')
_, scale = common.set_resized_input(interpreter, image.size, lambda size: image.resize(size, Image.LANCZOS))
interpreter.invoke()

detections = detect.get_objects(interpreter, scale)

The output error has changed now to following:

python tpu_detect.py 
Traceback (most recent call last):
  File "/home/pi/tpu_detect.py", line 13, in <module>
    detections = detect.get_objects(interpreter, scale)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/yolo_env/lib/python3.11/site-packages/pycoral/adapters/detect.py", line 214, in get_objects
    elif common.output_tensor(interpreter, 3).size == 1:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/yolo_env/lib/python3.11/site-packages/pycoral/adapters/common.py", line 29, in output_tensor
    return interpreter.tensor(interpreter.get_output_details()[i]['index'])()
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

To round it up, I ran a pip freeze and reduced the output to what I thought was relevant:

flatbuffers==24.3.25
h5py==3.10.0
keras==2.13.1
matplotlib==3.8.4
numpy==1.24.3
onnx==1.16.0
onnx-graphsurgeon==0.3.27
onnx2tf==1.17.5
onnxruntime==1.17.1
onnxsim==0.4.36
opencv-python==4.9.0.80
pandas==2.2.1
pillow==10.3.0
protobuf==4.25.3
pycoral @ file:///home/pi/pycoral-2.13.0-cp311-cp311-linux_aarch64.whl
python-dateutil==2.9.0.post0
PyYAML==6.0.1
tensorboard==2.13.0
tensorboard-data-server==0.7.2
tensorflow-aarch64==2.13.1
tensorflow-estimator==2.13.0
tensorflow-io-gcs-filesystem==0.36.0
tflite-runtime==2.14.0
tflite-support==0.1.0a1
torch==2.2.2
torchvision==0.17.2
tqdm==4.66.2
ultralytics==8.1.43

Not sure, which versions are really required here. Is there a list to check?

[pderrenger]

@StefansAI hey there! 🌞 Great to hear you've made progress with TensorFlow 2.13.1 on your Rpi5. It looks like you're encountering some final hurdles related to the Edge TPU compiler, which indeed can be a bit tricky since the compiler for ARM architecture isn’t readily available. Your detailed documentation and adaptation process are commendable! 🚀

Regarding the KeyError: 'output_0' and your following IndexError, these suggest there might be a disconnect between the expected model output tensor names and what's being provided. Since pycoral and the Edge TPU compiler expect specific outputs, ensuring the exported .tflite model aligns with these expectations is critical.

For the Edge TPU, TensorFlow Lite models need to be fully quantized, and it seems like you're on the right track with quantization. If you're still facing issues, one workaround could involve double-checking the model output names before and after quantization to ensure they match pycoral's expectations.

Since the Edge TPU compiler installation directly on the Raspberry Pi is a challenge, your approach of exporting the model elsewhere and transferring it is wise. If anyone has found an innovative solution or workaround for the Edge TPU compiler on ARM architecture, this is a valuable opportunity for them to share insights.

For the Pip freeze and library version concerns, the relevant ones mostly align with TensorFlow Lite runtime and pycoral versions. Given your setup, you might not need TensorFlow if you're running inferences directly with TensorFlow Lite runtime (tflite-runtime). This could simplify your environment a tad more.

In essence, focusing on the compatibility of exported model tensor names with the pycoral adapter and ensuring all model operations are Edge TPU compatible through careful quantization should be the priority. If the community has additional insights or solutions, chiming in would greatly benefit everyone working on similar projects. Best of luck, and keep pushing the boundaries! 🌟

[DAVIDNYARKO123]

I encountered similar challenges and ultimately developed a "solution". I contribute to a PyPI package designed to support the Ultralytics community, specifically for Raspberry Pi users employing the EDGE TPU USB Accelerator. This package simplifies the setup process by ensuring the correct versions of necessary runtimes are installed with just a single pip and setup command. It also provides several default YOLO models for immediate download and experimentation.

I hope this information is useful to you.

GitHub:

https://github.com/DAVIDNYARKO123/edge-tpu-silva

PyPi:

https://pypi.org/project/edge-tpu-silva/

YouTube:

https://www.youtube.com/watch?v=sOxQTRRh9tw

I just posted the video above after seeing this post.

[DAVIDNYARKO123]

Hey there! 🌟 Thanks for sharing your work on the edge-tpu-silva package - it sounds like a fantastic resource for the community, especially for those working with Raspberry Pi and the Coral Edge TPU. Your effort to simplify the setup process and provide readily available YOLO models is much appreciated. 🙏

For anyone interested in leveraging the power of Coral's Edge TPU on a Raspberry Pi, this package could be a great starting point. I'll definitely check out your YouTube video for a deeper dive into how it all works. 📺💡

Please keep up the great work, and feel free to reach out if you think there are ways we could collaborate or further support your efforts. The link to our guide on using Ultralytics YOLOv8 with a Coral Edge TPU on a Raspberry Pi might also complement your solution: Coral Edge TPU on Raspberry Pi with Ultralytics YOLOv8.

Best of luck with your project, and looking forward to seeing more from you! 🚀

Hello @pderrenger
Thank you for the encouraging words! ✨I’m glad you found the edge-tpu-silva package interesting and I’m eager to explore the guide you provided. It was a great complement and inspiration to this work.

I’m open to collaborating and would love to discuss how we can support this project. Looking forward to connecting soon! 🚀

[StefansAI]

Hi David,

I tried your approach. I always start from a new SD card image of the 64bit Bookworm in the RPi5 and then run the suggested commands. In your case it is:

python3 -m venv .venv
source .venv/bin/activate
pip install edge-tpu-silva
silvatpu-linux-setup

Then I downloaded the first of your exported models and run this line:

silvatpu -p det -m Downloads/240_yolov8n_full_integer_quant_edgetpu.tflite -i /dev/video0 -z 240 -t 0.5 -v True

Then I get this error:

(.venv) pi@pi5:~ $ silvatpu -p det -m Downloads/240_yolov8n_full_integer_quant_edgetpu.tflite -i /dev/video0 -z 240 -t 0.5 -v True
Traceback (most recent call last):
  File "/home/pi/.venv/lib/python3.11/site-packages/ultralytics/nn/autobackend.py", line 321, in __init__
    from tflite_runtime.interpreter import Interpreter, load_delegate
ModuleNotFoundError: No module named 'tflite_runtime'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/.venv/bin/silvatpu", line 8, in <module>
    sys.exit(silvatpu())
             ^^^^^^^^^^
  File "/home/pi/.venv/lib/python3.11/site-packages/edge_tpu_silva/__main__.py", line 97, in silvatpu
    for _, _ in outs:
  File "/home/pi/.venv/lib/python3.11/site-packages/edge_tpu_silva/silva/silva_detect.py", line 38, in process_detection
    outs = model.predict(
           ^^^^^^^^^^^^^^
  File "/home/pi/.venv/lib/python3.11/site-packages/ultralytics/engine/model.py", line 445, in predict
    self.predictor.setup_model(model=self.model, verbose=is_cli)
  File "/home/pi/.venv/lib/python3.11/site-packages/ultralytics/engine/predictor.py", line 297, in setup_model
    self.model = AutoBackend(
                 ^^^^^^^^^^^^
  File "/home/pi/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/.venv/lib/python3.11/site-packages/ultralytics/nn/autobackend.py", line 323, in __init__
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

I then tried to install tensorflow-aarch64==2.13.1, but ran into problems with dependencies of h5py. Any thoughts?

[DAVIDNYARKO123]

Hi @StefansAI
I will be glad to assist. Please what was the terminal output after running the setup command:

silvatpu-linux-setup

Did you see a message indicating successful installation, including the runtime version, at the end of the setup process?

Please list your Python version.

Be sure to use the recommended Python versions. As well as your apt-get is updated and upraded

sudo apt-get update
sudo apt-get upgrade

[Decoo13]

Hey David, thanks again for the information!

everything went smoothly for me I followed the README instructiions and everything worked well on my end.
I would like to note that I did place everything in the home directory to make things simpler in terms of placing the correct path for each of the files.

[pderrenger]

Hey David,

That's fantastic to hear! 🚀 Glad the setup went smoothly for you. Placing everything in the home directory can indeed simplify managing paths, especially for newcomers. It's a practical tip that could benefit many users, so thanks for highlighting it!

For anyone else following this thread, remember to check the Raspberry Pi model and OS compatibility, especially for tasks involving hardware accelerators like the Coral Edge TPU. Our documentation provides a detailed guide for setups like these, ensuring you get the best performance out of your Edge TPU and Raspberry Pi combo.

Keep exploring and sharing your insights, David! If you've got more questions or run into hiccups, feel free to drop them here. 🌟

[torayeff]

Is this also supported for M.2 Accelerator A+E key instead of USB? I am currently connecting directly to that interface on my RPi 5.

[pderrenger]

Hey there! 👋 The direct support for the M.2 Accelerator with the A+E key on Raspberry Pi isn’t explicitly detailed in our documentation. However, as long as your Raspberry Pi 5 can recognize the accelerator as a compatible device and you have the proper drivers/runtime installed, our guide for using the Coral Edge TPU with a Raspberry Pi should still be applicable. The main steps involve setting up the Edge TPU runtime correctly and exporting your model to a format compatible with the Coral Edge TPU (i.e., TensorFlow Lite with full integer quantization).

Here’s a quick snippet on how to export your model for the Edge TPU:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/model.pt')  # Replace with your model path

# Export the model for Edge TPU
model.export(format='edgetpu')

Make sure the Edge TPU runtime is installed properly on your RPi, and you should be good to go. If you encounter specific issues with the M.2 interface, it might be worth checking on compatibility forums or the official Coral documentation. 😊

[StefansAI]

Hi @pderrenger:
I gave it one more shot to install and run your libraries for Python3.11 as described here together with Pycoral for 3.11 from Oberluz. But in the end, there is still that mismatch between interpreter and detect.get_objects().
interpreter.get_output_details() returns a list with only one element, but the element of the index = 3 is requested. I tried to debug with VSS code, but there was nothing I else I could see. I have a screenshot, but I don't see how to attach it here.

Hi @DAVIDNYARKO123:
As said, I tried the bookworm out of the box with Python 3.11 and was really hoping I get it with 3.11 to work. I saw now, you required <3.10. So, I installed 3.9 in an venv and followed your instructions. It works and I get text outputs, no video or images. I can dig into that to see something. Thanks, I will check it out more.

[DAVIDNYARKO123]

@StefansAI
Great, I am glad 🎉!
To get the frame outputted, you can set the —-show/-s to True.

You can refer to the git link for more on that.

Future version of edge-tpu-silva which i am currently working on will include 3.10 and 3.11.

[StefansAI]

@DAVIDNYARKO123, thanks, I found it, started to play around with the code and added displaying the inference times.
With 640 yolov8n export and camera input the times are around 240ms average. My own test video with only 4 classes is around 210ms. For comparison, my Jetson Nano with only 4 classes does the same around 70ms.
With your 240 exports, the inference time on life stream video drops to ~30ms, which sounds great.

[StefansAI]

@DAVIDNYARKO123:
I had to realize, that it doesn't use the TPU at all and the acceleration of less than factor 2 compared to yolo without TPU must have come from the integer export. While streaming from the camera I unplugged the TPU and it kept running with the same speed. Can you confirm?

[StefansAI]

Hi @oliverlibaw, @pderrenger, @DAVIDNYARKO123

First of all, after some debugging, I had to find out, that there is an implicit name convention mentioned nowhere. Here is the export as recommended:

yolo export model=yolov8n.pt format=edgetpu

Despite the format= edgetpu, this export function creates file names like this for instance:

yolov8n_integer_quant.tflite

But this will never be directed to the TPU. To run it on the TPU, it has to have a name with "_edgetpu.tflite" in the name, like

yolov8n_integer_quant_edgetpu.tflite

So, you'll have to rename all exported files or make a copy to that name to compare both.
I tried to follow this article and also installed the edge_tpu_silva-version in a python 3.9 venv for comparison. Overall, both installation test results were the same and disappointing. There is a speed increase from *.pt models to *.tflite models through the optimization, but no difference between CPU and TPU execution. For 640 size the TPU execution even exceeded the CPU time.

Image Size	Model	RPi5-CPU/TPU	RPi5-Execution	Jetson Nano Execution
640	*.pt	CPU	385.2ms	64.4ms
	*int8.ftlite	CPU	380.3ms
	*int8_edgetpu.ftlite	TPU	383.3ms
320	*.pt	CPU	110.5ms	38.5ms
	*int8.ftlite	CPU	68.4ms
	*int8_edgetpu.ftlite	TPU	68.2ms
224	*.pt	CPU	64.9ms	37.9ms
	*int8.ftlite	CPU	27.6ms
	*int8_edgetpu.ftlite	TPU	27.7ms

To verify that the edgetpu version is really executed on the TPU, I unplugged the USB cable and observed the USB transfer error and the immediate abortion of the execution. The non-edgetpu model kept running when unplugging, so it must have been executed on the CPU only.

See my scripts here: Yolov8_Rpi5_CoralUSB

[oliverlibaw]

Thanks so much @StefansAI.

[UsamaRehman001]

@StefansAI how do you give camera source to the model. or frame by frame of 640 x 640. because even when i export the model. some layers are not quantized for the edgetpu shown by log file. also i get segmentation fault for any size greater than 192. can you help me figure out

[StefansAI]

@UsamaRehman001

Look for predict examples like here for instance here:

model = YOLO("model.pt")
# accepts all formats - image/dir/Path/URL/video/PIL/ndarray. 0 for webcam
results = model.predict(source="0")

source=0 is the webcam (USB). stream=True and show=True

[UsamaRehman001]

@StefansAI my camera is CSI camera. thats why i am confused for the source. thats a separate issue but any model exported for greater than 192x192 give "Segmentation Fault"

[FearL0rd]

Hello my friends from Earth.
This is the compiler for ARM
https://github.com/FearL0rd/edgetpu_compiler_for_ARM64

[glenn-jocher]

@FearL0rd hello! 🌍 Thanks for sharing the ARM compiler link for the Edge TPU. It's definitely useful for those looking to run inference on ARM64 devices with Edge TPUs.

For incorporating it with Ultralytics YOLOv8, you would generally export the YOLO model to a TensorFlow Lite format that's compatible with the Edge TPU. Just make sure the exported model is fully quantized to leverage the Edge TPU's capabilities. Here's how you might export the model using the Ultralytics framework:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/model.pt')

# Export the model to a TFLite format compatible with Edge TPU
model.export(format='tflite', int8=True)

Then, you can compile this .tflite model for the Edge TPU using the ARM-compatible compiler that you've mentioned. If there's anything specific you need help with regarding this process, feel free to ask! 🚀

[StefansAI]

@glenn-jocher, @FearL0rd,

I downloaded the compiler, did a new export with the code above, but get an error from the compiler when using

./edgetpu_compiler model_full_integer_quant.tflite

Internal compiler error. Aborting!

Other export outputs give "Model not quantized". Any thoughts, why the fully quantized model throws an error?

[glenn-jocher]

To fix the "Internal compiler error. Aborting!" when running edgetpu_compiler:

1. Update the Compiler: Ensure you are using the latest version of the Edge TPU compiler. You can download it from the official Coral website.

2. Check Model Compatibility: Verify that your model is fully integer-quantized and compatible with Edge TPU. Ensure all operations in the model are supported by the TPU.

3. Simplify the Model: Try reducing the model complexity by removing or simplifying unsupported operations.

4. Check for File Corruption: Ensure the .tflite model file is not corrupted.

5. Run on a Different Machine: Sometimes, trying the compilation on a different machine or environment can help identify if the issue is system-specific.

6. Consult Logs: Look at the detailed logs or error messages provided by the compiler for more specific issues.

By following these steps, you should be able to resolve or identify the cause of the internal compiler error.

[ninopleno]

Hi there everyone!

Any stable solution for using Coral Accelerator USB?

Thanks in advanced!

[pderrenger]

@ninopleno hello!

Yes, YOLOv8 is compatible with the Picamera2 for Raspberry Pi. Your code looks well set up for integrating with the Coral Edge TPU for enhanced performance. If you're using the YOLOv8 model, ensure that it's properly converted to the TensorFlow Lite format compatible with the Edge TPU as detailed in our Coral Edge TPU guide.

Just a quick note: make sure to handle the draw_objects function correctly to visualize the results on your frames. If you encounter any specific issues or have further questions, feel free to ask!

Happy coding! 😊

[ninopleno]

Yes, I encountered 1 specific issue: The "libedgetpu.so.1" can apparently never be found. I install tflite-runtime using

python -m venv --system-site-packages env

Because I am installing everything using sudo apt install rather than with pip install as stated in the official Picamera 2 manual.

Any idea?

[pderrenger]

Hey there!

It sounds like the system can't locate the libedgetpu.so.1 library. This issue might be due to the library not being in the expected directory or not being linked correctly. Here’s a quick fix you can try:

1. Check Installation: Ensure that the Edge TPU runtime is correctly installed and that libedgetpu.so.1 exists in /usr/lib/aarch64-linux-gnu/. You can check this with:

   ls /usr/lib/aarch64-linux-gnu/libedgetpu.so.1

2. Set Library Path: If the library is in the correct place but still not found, you might need to explicitly set the library path when running your Python script:

   export LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu:$LD_LIBRARY_PATH
   python your_script.py

3. Reinstall the Runtime: If the file doesn’t exist, try reinstalling the Edge TPU runtime:

   sudo dpkg -i path/to/libedgetpu-runtime.deb

Make sure to replace path/to/libedgetpu-runtime.deb with the actual path to the downloaded .deb file suitable for your Raspberry Pi OS version.

Hope this helps! Let me know if there's anything else you need. 😊

[ninopleno]

Now I downgraded the OS to Bullseye (with desktop) and both versions, the Ultralytics and tflite-runtime, give me this error message.

FPS: N/A
Loading src/yolov8n_full_integer_quant_edgetpu.tflite for TensorFlow Lite Edge TPU inference...
Traceback (most recent call last):
  File "/home/blackops2/HQ/src/ultra.py", line 40, in <module>
    main()
  File "/home/blackops2/HQ/src/ultra.py", line 30, in main
    results = model.predict(resized_frame)
  File "/home/blackops2/.local/lib/python3.9/site-packages/ultralytics/engine/model.py", line 446, in predict
    self.predictor.setup_model(model=self.model, verbose=is_cli)
  File "/home/blackops2/.local/lib/python3.9/site-packages/ultralytics/engine/predictor.py", line 297, in setup_model
    self.model = AutoBackend(
  File "/home/blackops2/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/blackops2/.local/lib/python3.9/site-packages/ultralytics/nn/autobackend.py", line 337, in __init__
    interpreter = Interpreter(model_path=w, experimental_delegates=[load_delegate(delegate)])
  File "/home/blackops2/.local/lib/python3.9/site-packages/tflite_runtime/interpreter.py", line 464, in __init__
    self._interpreter = _interpreter_wrapper.CreateWrapperFromFile(
ValueError: Missing registration for opcode_index 0
Missing registration for opcode_index 1
Missing registration for opcode_index 2
Missing registration for opcode_index 1
Missing registration for opcode_index 2
Missing registration for opcode_index 3
Missing registration for opcode_index 4
Missing registration for opcode_index 2
Missing registration for opcode_index 1
Missing registration for opcode_index 1
Missing registration for opcode_index 5
Missing registration for opcode_index 6
Missing registration for opcode_index 2
Missing registration for opcode_index 7
Missing registration for opcode_index 2
Missing registration for opcode_index 4
Missing registration for opcode_index 8
Missing registration for opcode_index 4
Missing registration for opcode_index 9
Missing registration for opcode_index 9
Missing registration for opcode_index 7
Missing registration for opcode_index 8
Missing registration for opcode_index 4
Missing registration for opcode_index 10
Missing registration for opcode_index 11
Missing registration for opcode_index 3

The model is quantized + compiled for edge tpu with tensorflow 2.13.x I used the notebook in this discussion to create the model.

Thank you very much in advanced for the fast reply. I have worked on this for 3 weeks now without any success 😅

[pderrenger]

@ninopleno hey there!

It looks like you're encountering a common issue related to missing operation registrations in TensorFlow Lite when using the Edge TPU. This typically happens when the model uses ops that are not supported by the TensorFlow Lite runtime configured for the Edge TPU.

Here are a couple of steps to potentially resolve this:

1. Ensure Compatibility: Double-check that all operations used in your model are supported by the TensorFlow Lite Edge TPU. You might need to modify your model to replace or remove unsupported operations.

2. Recompile the Model: Use the latest version of the Edge TPU Compiler to recompile your .tflite model. Sometimes, recompiling with the latest tools can resolve these issues as support for operations can improve over time.

3. TensorFlow Lite Runtime: Make sure you're using a compatible version of the TensorFlow Lite runtime. If there's a mismatch between the runtime and the model's requirements, that could cause the errors you're seeing.

For detailed steps on model conversion and setup, you might find our guide on Coral Edge TPU usage helpful. 😊

If the issue persists, please provide more details about the operations used in your model, and we can look into this further!