SAM (Segment Anything Model) - Ultralytics YOLOv8 Docs

[giscus[bot]]

SAM (Segment Anything Model) - Ultralytics YOLOv8 Docs

Learn about the Segment Anything Model (SAM) and how it provides promptable image segmentation through an advanced architecture and the SA-1B dataset.

https://docs.ultralytics.com/models/sam/

[Adeel-Mumtaz]

Hi, How to you prompt based version of SAM? is it enabled? can you post an example

[codename-cn]

from ultralytics import YOLO, Prompt
ImportError: cannot import name 'Prompt' from 'ultralytics'

[pderrenger]

Hey there! It looks like you're having trouble importing the Prompt class from the ultralytics package. The Prompt class is actually part of the ultralytics.models.sam module, not directly under ultralytics. Here's how you can import it correctly:

from ultralytics.models.sam import Prompt

Give this a try, and it should resolve the import error you're encountering. If you have any more questions or run into further issues, feel free to ask. Happy coding! 😊👨‍💻

[codename-cn]

@pderrenger still ImportError: cannot import name 'Prompt' from 'ultralytics.models.sam'

from ultralytics import SAM
from ultralytics.models.sam import Prompt

# Load the SAM model
model = SAM('path/to/sam_b.pt')

# Create a prompt
prompt = Prompt("a photo of a cat")

# Perform segmentation with the prompt
results = model.predict(source='image.jpg', prompt=prompt)

# The results object contains the segmentation masks and other details
masks = results.masks.data  # Segmentation masks

[mohabtarek96]

@pderrenger same here ImportError: cannot import name 'Prompt' from 'ultralytics.models.sam'

[pderrenger]

Hello @mohabtarek96! It seems there's been a misunderstanding regarding the Prompt class for the SAM model. Currently, the Ultralytics YOLOv8 library does not include a Prompt class for SAM or any other models. The SAM model is designed for image segmentation and can be used directly with image inputs to perform segmentation tasks.

Here's a basic example of how you can use the SAM model for segmentation:

from ultralytics import SAM

# Load the SAM model
model = SAM('sam_b.pt')  # Ensure you have the correct path to the model

# Perform segmentation on an image
results = model('path/to/your/image.jpg')

# Accessing the segmentation masks
masks = results.masks

Make sure to replace 'path/to/your/image.jpg' and 'sam_b.pt' with the actual paths to your image and SAM model file, respectively.

If you're looking to perform more complex tasks or integrate specific prompts, you might need to manually handle the inputs and outputs according to the model's capabilities. For detailed guidance and the latest updates, please refer to the official Ultralytics documentation. If you have any more questions or need further assistance, feel free to ask! 😊

[wydilearn]

Hi, How to train a yolov8 model based on the SAM model with my own dataset? Is there any document to explain it? Thank you.

[pderrenger]

@wydilearn to train a YOLOv8 model based on the Segment Anything Model (SAM) with your own dataset, you can follow the general training procedure for YOLOv8, with a few adjustments specific to segmentation tasks. Here's a concise guide:

1. Prepare Your Dataset: Format your dataset with segmentation labels. Your dataset should include images and corresponding segmentation masks.

2. Create a Data YAML: Make a .yaml file that specifies the paths to your training and validation data, the number of classes, and class names.

3. Select a SAM Model: Choose a SAM model configuration file or checkpoint. If you're starting from a pretrained SAM model, you can use its .pt file.

4. Train Your Model: Use the train mode with the SAM model and your data YAML file. Here's a simplified example using Python:

   from ultralytics import YOLO
   
   # Load a SAM model configuration or checkpoint
   model = YOLO('path/to/sam-model.yaml', pretrained='path/to/sam-model.pt')
   
   # Train the model on your dataset
   results = model.train(data='path/to/your-data.yaml', epochs=50)

5. Monitor Training: Keep an eye on the training process and adjust hyperparameters if necessary.

6. Validate Your Model: After training, validate your model to ensure it performs well on unseen data.

   # Validate the model
   results = model.val(data='path/to/your-data.yaml')

For detailed instructions and explanations, please refer to the Ultralytics documentation on training models. You can find comprehensive guides and examples that will help you through the process. If you encounter any issues or have specific questions, feel free to open an issue in the YOLOv8 repo for support. 🚀📚

[TY-teo]

Unbelievable!

[pderrenger]

@TY-teo thank you! The credit goes to the entire YOLO community and the Ultralytics team for their hard work and dedication. If you have any questions or need assistance with YOLOv8 or SAM, feel free to reach out. Happy coding! 😊🚀

[mattmazzola]

Are there code samples for predictions using SAM?

I tried adapting the samples for YOLOv8 but it seems SAM is different.

from ultralytics import SAM

sam_model = SAM('sam_b.pt')
sam_model.info()

sam_results = sam_model.predict(
    'https://ultralytics.com/images/bus.jpg',
    save=True,
    save_crop=True)

print(sam_results)

Error
Seems the model didn't detect any segments and boxes is not defined?

/local_disk0/.ephemeral_nfs/envs/pythonEnv-ac9280b0-c95c-431f-b4d5-f21d5e128f9e/lib/python3.8/site-packages/ultralytics/yolo/engine/results.py in save_crop(self, save_dir, file_name)
    312         if isinstance(file_name, str):
    313             file_name = Path(file_name)
--> 314         for d in self.boxes:
    315             save_one_box(d.xyxy,
    316                          self.orig_img.copy(),

TypeError: 'NoneType' object is not iterable

Any suggestions?

[pderrenger]

@mattmazzola it looks like you're trying to use the SAM model for image segmentation, which is a bit different from the typical object detection models like YOLOv8. The SAM model does not produce bounding boxes (boxes) but rather segmentation masks. Therefore, you should be accessing the segmentation results differently.

Here's a corrected version of your code snippet that should work for SAM:

from ultralytics import YOLO  # SAM is part of the YOLO package

# Load the SAM model
sam_model = YOLO('sam_b.pt')

# Perform segmentation on an image
sam_results = sam_model.predict('https://ultralytics.com/images/bus.jpg')

# Print the results
print(sam_results)

# If you want to save the results, you can do so by accessing the masks
# and using the appropriate method to save them.
if sam_results.masks is not None:
    for mask in sam_results.masks.data:
        # Here you can save each mask individually
        # For example, you could use OpenCV to save the mask image
        pass

Please note that the predict method for SAM will return segmentation masks, which you can access through sam_results.masks. The boxes attribute you were trying to access is specific to object detection and does not apply to segmentation models.

If you want to save the segmentation results, you will need to handle the masks specifically. The save_crop parameter is not applicable for segmentation results as it is intended for saving cropped images based on bounding boxes, which are not produced by SAM.

For more detailed information on using SAM and handling segmentation results, please refer to the Segmentation Task Documentation. 📚

[danielzhangau]

Which tools that I can use to label the SAM auto generated labels?

[LeSoleil1]

cvat.ai would be a good tool.

[pderrenger]

@LeSoleil1 for refining SAM auto-generated labels, you can use annotation tools that support segmentation tasks. While I can't provide direct links, popular tools like CVAT, LabelMe, and VGG Image Annotator (VIA) are commonly used in the community for such tasks. These tools offer functionalities to edit and correct segmentation masks, which can be helpful in improving the quality of auto-generated labels. 🎨🛠️

[Nimgwen]

how can I plot some of the generated masks on top of the images? Similar to how you can plot the predicted bounding boxes using YOLOv8.

[pderrenger]

@Nimgwen to plot the generated masks on top of the images using YOLOv8, you can follow these steps after performing inference with a segmentation model like SAM:

1. Perform inference to get the results object, which contains the predicted masks.
2. Access the results.masks attribute to retrieve the segmentation masks.
3. Use a visualization library like Matplotlib to overlay the masks on the images.

Here's a concise example of how you might do this:

from ultralytics import YOLO
import matplotlib.pyplot as plt
import numpy as np

# Load your trained segmentation model
model = YOLO('path/to/your/sam-model.pt')

# Perform inference on an image
results = model('path/to/your/image.jpg')

# Get the original image as a numpy array
img = np.array(results.imgs[0])

# Get the segmentation masks
masks = results.masks.data

# Plot the image and overlay the masks
plt.imshow(img)
for mask in masks:
    plt.imshow(mask, alpha=0.5)  # Adjust alpha for mask transparency
plt.axis('off')  # Hide the axes
plt.show()

This code snippet assumes you have a trained SAM model and an image to perform inference on. The alpha parameter in plt.imshow() controls the transparency of the masks. Adjust it as needed to visualize the masks clearly on top of the image. 🎨

[Xyihang]

Auto-Annotation:
i hope we can ues SAM to label data for yolo,thx!!!!!

[pderrenger]

@Xyihang absolutely! The Segment Anything Model (SAM) can indeed be used to generate annotations for your datasets. SAM's advanced segmentation capabilities can help pre-label your data, which you can then refine and use to train YOLO models. This can significantly speed up the data preparation process. If you have any further questions or need assistance with using SAM for auto-annotation, feel free to ask! 😊🚀

[LLsmile]

How to transform it into onnx format and run in c++?

[pderrenger]

@LLsmile to export a YOLOv8 model to ONNX format and run it in C++, follow these steps:

1. Export to ONNX:
   Use the export method with the format='onnx' argument. Ensure you have the model .pt file ready.

   from ultralytics import YOLO
   
   # Load your trained model
   model = YOLO('path/to/your/model.pt')
   
   # Export the model to ONNX format
   success = model.export(format='onnx')

2. Run in C++:
   Once you have the ONNX model, you can use the ONNX Runtime C++ API to perform inference. Here's a simplified example of how you might do this:

   #include <onnxruntime/core/session/onnxruntime_cxx_api.h>
   
   int main() {
       // Initialize ONNX Runtime
       Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "YOLOv8");
   
       // Set up options for inference session
       Ort::SessionOptions session_options;
       session_options.SetIntraOpNumThreads(1);
   
       // Load the ONNX model
       const char* model_path = "path/to/your/model.onnx";
       Ort::Session session(env, model_path, session_options);
   
       // Define input and output nodes
       Ort::AllocatorWithDefaultOptions allocator;
       char* input_name = session.GetInputName(0, allocator);
       char* output_name = session.GetOutputName(0, allocator);
   
       // Prepare input tensor
       std::vector<float> input_tensor_values = ...; // Fill with your input data
       std::vector<int64_t> input_shape = ...; // Define the shape of your input tensor
       Ort::Value input_tensor = Ort::Value::CreateTensor<float>(
           allocator, input_tensor_values.data(), input_tensor_values.size(), input_shape.data(), input_shape.size());
   
       // Run inference
       auto output_tensors = session.Run(Ort::RunOptions{nullptr}, &input_name, &input_tensor, 1, &output_name, 1);
   
       // Process output
       float* floatarr = output_tensors.front().GetTensorMutableData<float>();
       // ... process your output data
   
       // Clean up
       allocator.Free(input_name);
       allocator.Free(output_name);
   
       return 0;
   }

3. Compile and Run:
   Compile your C++ code with the appropriate ONNX Runtime libraries linked. The exact compilation command will depend on your setup and the build system you're using.

Remember to handle pre-processing and post-processing steps as required by the model, such as image resizing, normalization, and interpreting the output tensors to obtain the final detection results.

For more detailed information on exporting models and running inference in C++, refer to the Ultralytics Docs and the ONNX Runtime documentation.

[Muhammad-Kaleem-Ullah]

Auto Generation of masks does not seem to be working!!! Similiar to .predict() it returns probabilities with bounding boxes. I'm expecting it to return segmented bounding boxes. Unlike FastSAM, it does not support segment_everything()

[pderrenger]

@Muhammad-Kaleem-Ullah i'm sorry to hear you're experiencing issues with the segmentation functionality. The SAM (Segment Anything Model) is designed to provide segmentation capabilities, and if you're not seeing segmented bounding boxes, there might be a few things to check:

1. Ensure you're using the correct model that supports segmentation. For instance, yolov8n-seg.pt is a model that includes segmentation capabilities.

2. Verify that you're calling the correct method for segmentation. If you're using .predict(), make sure to set the task parameter to 'segment' to activate the segmentation mode.

3. Check that your environment is set up correctly and that you have the latest version of the YOLOv8 repository, as updates may include fixes and improvements for segmentation.

4. Review the documentation for any additional parameters that might need to be set for segmentation to work as expected.

If you've checked these points and the issue persists, please open an issue on the YOLOv8 repository with details of your code and the steps you've taken. This will allow us to investigate further and assist you in resolving the problem. 🛠️

[tolahs]

Hi everybody!
I want to ask sth. for this link topic given below
Auto-Annotation: A Quick Path to Segmentation Datasets

What is the correct choose if i have with custom pre-trained model with from created createt my custom dataset

Use my custom detection model(my_custom_detection_model) or use yolov pretrained model(yolov8x.pt) ?

auto_annotate(data="path/to/images", det_model="yolov8x.pt", sam_model='sam_b.pt')
auto_annotate(data="path/to/images", det_model="my_custom_detection_model.pt", sam_model='sam_b.pt')

[pderrenger]

@tolahs when deciding between using your custom detection model or a pretrained YOLOv8 model for auto-annotation, consider the following:

• If your custom model is well-trained on a dataset similar to your target images and performs with high accuracy, it might be more beneficial to use your custom model for auto-annotation.
• If your custom model's performance is not satisfactory or if it's not trained on a relevant dataset, using the pretrained yolov8x.pt might yield better results, as it has been trained on a diverse and large dataset.

In summary, choose the model that has the best performance on the type of images you are working with. If you're unsure, you could experiment with both and compare the quality of the auto-annotations. Here's how you can use both options:

# Using a pretrained YOLOv8 model
auto_annotate(data="path/to/images", det_model="yolov8x.pt", sam_model='sam_b.pt')

# Using your custom detection model
auto_annotate(data="path/to/images", det_model="my_custom_detection_model.pt", sam_model='sam_b.pt')

Remember to evaluate the auto-annotated results to ensure they meet your quality standards for training your segmentation models. 🎯🔍

[tolahs]

thanks for long explained answer. I will try and compare results as you said.

[glenn-jocher]

@tolahs you're welcome! That sounds like a great plan. Comparing the results will definitely help you choose the best model for your needs. If you have any more questions or need further assistance along the way, feel free to reach out. Good luck with your auto-annotation! 👍🚀

[deepali-ds]

How to extract bounding boxes for multiple images to pass it to SAM model for segmentation?

model = YOLO('/content/drive/MyDrive/yolo/runs_yolov8_500epochs/detect/train/weights/best.pt')
results= model.predict(source ='/content/drive/MyDrive/yolo/val/images', conf=0.5)

[glenn-jocher]

Hi there! It looks like you're trying to load a fine-tuned SAM model but are facing some issues with the method provided in the documentation.

For loading a SAM model the correct way is:

from ultralytics import YOLO  # SAM uses the same interface as YOLO

# Load the SAM model
sam_model = YOLO('path/to/your/sam_b.pt')

Make sure the path to your model file 'sam_b.pt' is correct and accessible. This should resolve the issue you're facing with loading the model. Let me know if you need further assistance! 😊

[deepali-ds]

error; AttributeError Traceback (most recent call last)
in <cell line: 15>()
13
14 # Iterate through images and their detections
---> 15 for img, result in zip(yolo_results.files, yolo_results):
16 # Prepare bounding boxes in the format expected by SAM
17 bboxes = result.boxes.xyxy[0] # Assuming each image from predict returns single tensor

AttributeError: 'list' object has no attribute 'files'

[glenn-jocher]

Hi there! It looks like you're encountering an attribute error because yolo_results is a list of Result objects, which doesn't have a .files attribute. Here’s how you should modify your loop to iterate through the results for multiple images:

from ultralytics import YOLO

# Load your YOLO model
model = YOLO('/content/drive/MyDrive/yolo/runs_yolov8_500epochs/detect/train/weights/best.pt') 

# Predict multiple images in a directory
results = model.predict(source='/content/drive/MyDrive/yolo/val/images', conf=0.5)

# Iterate through results and process each
for result in results:
    # Get bounding boxes in xyxy format
    boxes = result.boxes.xyxy
    # Process each bbox or pass it to another model etc.
    print(boxes)

This should correctly loop through each image result and allows you to access the bounding boxes directly from each Result object. If you need further customization or assistance, feel free to ask! 😊

[deepali-ds]

Thank you for your reply. I could get bounding boxes now and I am passing those bboxes to SAM model;
sam_results = sam_model(source='/content/drive/MyDrive/val/images', bboxes=boxes.cpu().numpy().tolist())
I am wondering what size images SAM model accept?
My images are 256x256

[glenn-jocher]

Hi there! Great to hear you've successfully harnessed the bounding boxes and are now experimenting with SAM! 🌟 To answer your question, SAM models, similar to many Ultralytics models, typically expect images resized to 640x640 pixels for optimal performance. If your images are 256x256, consider resizing them to 640x640 before passing them through SAM. This resizing step can help in maintaining consistency with how the model was trained, thereby potentially improving the output quality.

Here's a quick way to resize your images:

import cv2

# Assuming 'img' is your 256x256 image
resized_img = cv2.resize(img, (640, 640))
sam_results = sam_model(source=resized_img, bboxes=boxes.cpu().numpy().tolist())

Let's optimize and get the most out of SAM! If you need further adjustments or have more queries, feel free to drop a message! 🚀

[rohinr7]

Dear Ultralytics,

I am using yolov8 detection model with xai (eigencam) its possible to segment the object (I detect using the yolov8n detection model), with this SAM and attribution weights i get ?

Thanks in advance

[pderrenger]

@rohinr7 hello!

Currently, our YOLOv8 detection models and SAM (Segment Anything Model) operate separately under our architecture. The YOLOv8n detection model excels in object detection tasks but does not perform instance segmentation, which is the functionality provided by SAM.

For your case, if you wish to segment objects that you've detected using YOLOv8n, incorporating SAM would be a suitable solution. Here’s a small example of how you can first detect objects with YOLO and then use SAM for segmentation:

from ultralytics import YOLO, SAM

# Load the detection model
yolo_model = YOLO('yolov8n.pt')

# Detect objects in an image
detections = yolo_model('path/to/your/image.jpg')

# Load the segmentation model
sam_model = SAM('sam_b.pt')

# Use detections to define prompts (like bounding boxes) for SAM
bboxes = [det.xywh for det in detections]  # Assuming the detections provide xywh format
segmented_output = sam_model('path/to/your/image.jpg', bboxes=bboxes)

Please adjust the model paths and image paths as per your configuration. This approach allows YOLOv8n to handle the detection, while SAM handles the segmentation based on the results from YOLOv8n.

Let me know if you have further questions! 😊

[rohinr7]

Thanks for the information and my another question is can i generate mask from the XAI output (look at the picture i shared you) that picture is a attribution method where the highlighted region is the prediction from yolov8 can i generate mask from that?

[pderrenger]

Hi there!

Generating a mask from the XAI output specifically, where regions are highlighted based on the model’s predictions (like shown in your image from YOLOv8), isn't directly supported as a feature in YOLO models or the SAM model.

However, you can approximate this by using a thresholding approach on the heatmap generated by the XAI (e.g., eigencam). Here’s a basic approach using Python:

import cv2
import numpy as np

# Assuming 'heatmap' is your XAI heatmap output as a numpy array
_, thresh = cv2.threshold(heatmap, threshold_value, 255, cv2.THRESH_BINARY)

threshold_value in the above code would be set based on your specific needs (typically a value that separates the 'important' parts from the 'unimportant').

For actual segmentation tasks, SAM would be the appropriate model, but it operates based on image data and certain prompts, not directly on heatmaps.

Let me know if you need further details on anything specific! 😊