Lag when adding small scatters to large heatmap animations

Start by running the supplied plotly.graph_objects.Figure example and compare animation with ADD_BBOXES enabled, disabled, and hidden through the updatemenus controls. Trace the Heatmap and Scatter frame-update behavior to identify why an inactive bbox still affects redraw time; done means large heatmap frames switch smoothly with bbox traces present.

I've noticed that adding a very small scatter to a large animation will make the lag when moving between frames very bag when working on a project requiring visualization of bboxes of segmentation outputs. If I don't have the bboxes then then there is no lag after startup. With bboxes it becomes very noticeable that the rerender is happening, and as the size of the heatmap increases it becomes worse. Is there a way to make it more smooth?

ADD_BBOXES=True
import numpy as np
import plotly.graph_objects
mock_segmentation_1 = plotly.graph_objects.Heatmap(
    z=np.random.randint(0, 100, (2000,512)),
)
mock_segmentation_2 = plotly.graph_objects.Heatmap(
    z=np.random.randint(0, 2, (2000,512)),
)
mock_bbox_1 = plotly.graph_objects.Scatter(
                                x=[100,100,400,400],
                                y=[200,500,500,200],
                                fillcolor="rgba(255,255,255)",
                                fill="toself",
                                hoveron="fills",
                                hoverinfo="text",
                                text='bbox1',
                                mode="lines",
                                showlegend=False,
                            )
mock_bbox_2 = plotly.graph_objects.Scatter(
                                x=[100,100,2000,2000],
                                y=[2000,500,500,2000],
                                fillcolor="rgba(255,255,255)",
                                fill="toself",
                                hoveron="fills",
                                hoverinfo="text",
                                text='bbox1',
                                mode="lines",
                                showlegend=False,
                            )


data1 = [mock_segmentation_1]
data2 = [mock_segmentation_2]
if ADD_BBOXES:
    data1 = [mock_segmentation_1, mock_bbox_1]
    data2 = [mock_segmentation_2, mock_bbox_2]


frames = [{'data': data1, 'name': '1'}, {'data': data2, 'name': '2'}]
figure_dict = {
    "data": frames[0]["data"],
    "frames": frames,
    "layout": {
        "sliders": [
            {
                "steps": [
                    {
                        "method": "animate",
                        "label": f"Frame {fname}",
                        "args": [
                            [fname],
                            {
                                "mode": "immediate",
                                "frame": {"redraw": True},
                                "transition": {'duration': 0},
                            },
                        ],
                    }
                    for fname in ['1', '2']
                ]
            }
        ],
    },
}
plotly.graph_objects.Figure(figure_dict)

https://github.com/user-attachments/assets/5a7af1d5-6afc-40ce-b45d-f965507c79bd

https://github.com/user-attachments/assets/16fd1a62-64bd-4d6d-882e-3d46189bc224

I tried to fix this by turning off the visibility of the scatter frames, adding this to the layout. However, the lag persists as long as the bboxes exist, even if they are not visible.

"updatemenus": [
            {
                'buttons': [
                    {
                        'args': [{'visible': [True, True]}],
                        'label': 'Enable BBoxes',
                        'method': 'update'
                    }, {
                        'args': [{'visible': [True, False]}],
                        'label': 'Disable BBoxes',
                        'method': 'update'
                    }
                ]
            }
        ]