[CVAT integration] Use pixelwise masks, not polygons, for instance segmentation

Proposal Summary

Following the FiftyOne's document, I figured out that FiftyOne's instance segmentation works with polygons rather than pixel-wise masked segments. For integrating CVAT and FiftyOne, instance segmentation would be much better when annotated with pixelwise masks rather than polygons.

Motivation

• What is the use case for this feature?

I am trying to integrate CVAT with FiftyOne, whilst using SAM for segmentation in CVAT.

• Why is this use case valuable to support for FiftyOne users in general?

This issue has already been suggested in https://github.com/voxel51/fiftyone/issues/3750. It seems that many users want pixelwise masks rather than polygons for segmentation annotations.

• Why is this use case valuable to support for your project(s) or organization?

When annotating segmentation using SAM in CVAT, the annotation results are masked segments, not polygons. It is needed to load dataset and annotations with masks in FiftyOne, load the annotated dataset into CVAT to add/edit/delete some annotations, and load the modified annotations back into FiftyOne. Expected annotation type is masked segments, not polygons, but instance segmentation for integration CVAT and FiftyOne seems to work only with polygons which don't appear to be expected as pixelwise segments.

• Why is it currently difficult to achieve this use case?

When I try to load detections of FiftyOne into CVAT, all annotations are gone and only images are loaded in CVAT. Also when I annotate segmentation using SAM in CVAT and try to load the annotated image data into FiftyOne, again any annotations disappear and again soley images are shown in FiftyOne App. When I set label_type=instance as in the code line below

dataset.annotate(
    anno_key,
    label_field='ground_truth',
    label_type='instance',
    url='http://localhost:8080/'
)

then the annotations are loaded but in polygons, not in pixel-wise segments.

What areas of FiftyOne does this feature affect?

• App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Details

Expected implementations for instance segmentation with pixelwise segments:

dataset.annotate(
    anno_key,
    label_field='detections',
    label_type='instances',
    url='http://localhost:8080/'
)

Expected implementations for instance segmentation with polygons:

dataset.annotate(
    anno_key,
    label_field='detections',
    label_type='polygons',
    url='http://localhost:8080/'
)

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently
• Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• No. I cannot contribute this feature at this time