Out of control data transfer for assets

Start by examining the reported queued Statamic\Search\InsertMultipleJob payload and the asset-listing timeout on the AWS S3 container; compare behavior with the queue stopped, as the reporter did. The issue provides no reproduction steps, logs, or identified cause, so first establish which requests or jobs account for the S3 usage. Done would mean identifying and fixing the cause, with a reproducible test or diagnostic evidence.

Bug description

I'm not really sure what to call the title of the bug report because I'm still trying to figure it out. I'll try provide more details soon if I can figure more out but it's a very expensive thing to debug.

Somehow since upgrading to Statamic 6? The queue..? or something transferred 2,851.887 GB of data (2,8 Tb) that cost an extra $256 to our AWS bill. Additionally it did 12 million (PUT,COPY,POST or List) Requests which cost an additional $60.

It took a while to figure out how to read the bucket usage from Amazon S3 but each day it's been charging $18 of usage to my assets.llresearch.dev bucket which is the one I use exclusively for our Statamic CMS assets on the test website.

So I think something went wrong with the queue if it made 12 million requests? I shut down the queue this morning and now it's only charging me $3 today.

It seems like the Statamic\Jobs\HandleEntrySchedule job runs every minute, so I'm not sure if that could be related somehow and causing the 12 million requests?

There are also some jobs in the queue related to Search that reference assets on AWS, the Statamic\Search\InsertMultipleJob one... but I'm not sure if that would be pinging AWS and trying to find or create metadata?

In the example payload for that job you can see it's full of my assets like asset::aws::archive\/images\/101_cover.jpg or asset::aws::audio\/101\/lesson_1.mp3

 id = 190
 queue = default
 payload = {"uuid":"dba5ab36-2152-441d-982b-2e3e9d458e1x","displayName":"Statamic\\Search\\InsertMultipleJob","job":"Illuminate\\Queue\\CallQueuedHandler@call","maxTries":null,"maxExceptions":null,"failOnTimeout":false,"backoff":null,"timeout":null,"retryUntil":null,"data":{"commandName":"Statamic\\Search\\InsertMultipleJob","command":"O:33:\"Statamic\\Search\\InsertMultipleJob\":3:{s:4:\"name\";s:7:\"default\";s:6:\"locale\";N;s:9:\"documents\";O:31:\"Statamic\\Assets\\AssetCollection\":2:{s:8:\"\u0000*\u0000items\";a:100:{i:0;s:30:\"asset::assets::don-quixote.svg\";i:1;s:30:\"asset::assets::icons\/about.svg\";i:2;s:34:\"asset::assets::icons\/beginners.svg\";i:3;s:30:\"asset::assets::icons\/books.svg\";i:4;s:37:\"asset::assets::icons\/carlas-niche.svg\";i:5;s:35:\"asset::assets::icons\/channeling.svg\";i:6;s:34:\"asset::assets::icons\/community.svg\";i:7;s:32:\"asset::assets::icons\/courses.svg\";i:8;s:28:\"asset::assets::icons\/faq.svg\";i:9;s:29:\"asset::assets::icons\/home.svg\";i:10;s:33:\"asset::assets::icons\/lawofone.svg\";i:11;s:32:\"asset::assets::icons\/library.svg\";i:12;s:41:\"asset::assets::icons\/magnifying-glass.svg\";i:13;s:30:\"asset::assets::icons\/media.svg\";i:14;s:36:\"asset::assets::icons\/meditations.svg\";i:15;s:33:\"asset::assets::icons\/navigate.svg\";i:16;s:33:\"asset::assets::icons\/podcasts.svg\";i:17;s:31:\"asset::assets::icons\/search.svg\";i:18;s:32:\"asset::assets::icons\/sidebar.svg\";i:19;s:30:\"asset::assets::icons\/study.svg\";i:20;s:31:\"asset::assets::icons\/topics.svg\";i:21;s:32:\"asset::assets::icons\/updates.svg\";i:22;s:31:\"asset::assets::icons\/videos.svg\";i:23;s:34:\"asset::assets::icons\/wanderers.svg\";i:24;s:29:\"asset::assets::icons\/wiki.svg\";i:25;s:42:\"asset::assets::team-photo-with-unicorn.png\";i:26;s:20:\"asset::aws::404.html\";i:27;s:34:\"asset::aws::archive\/images\/101.jpg\";i:28;s:40:\"asset::aws::archive\/images\/101_cover.jpg\";i:29;s:62:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1024_x_768.jpg\";i:30;s:62:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1152_x_864.jpg\";i:31;s:63:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1280_x_1024.jpg\";i:32;s:63:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1600_x_1200.jpg\";i:33;s:61:\"asset::aws::archive\/images\/A_Wanderers_Handbook_800_x_600.jpg\";i:34;s:57:\"asset::aws::archive\/images\/A_Wanderers_Handbook_image.jpg\";i:35;s:35:\"asset::aws::archive\/images\/Vol1.jpg\";i:36;s:35:\"asset::aws::archive\/images\/Vol2.jpg\";i:37;s:56:\"asset::aws::archive\/images\/a_book_of_days_1024_x_768.jpg\";i:38;s:56:\"asset::aws::archive\/images\/a_book_of_days_1152_x_864.jpg\";i:39;s:57:\"asset::aws::archive\/images\/a_book_of_days_1600_x_1200.jpg\";i:40;s:55:\"asset::aws::archive\/images\/a_book_of_days_800_x_600.jpg\";i:41;s:51:\"asset::aws::archive\/images\/a_book_of_days_cover.jpg\";i:42;s:57:\"asset::aws::archive\/images\/a_book_of_days_cover_image.jpg\";i:43;s:57:\"asset::aws::archive\/images\/a_wanderers_handbook_cover.jpg\";i:44;s:35:\"asset::aws::archive\/images\/abod.jpg\";i:45;s:34:\"asset::aws::archive\/images\/ach.jpg\";i:46;s:39:\"asset::aws::archive\/images\/archives.jpg\";i:47;s:34:\"asset::aws::archive\/images\/awh.jpg\";i:48;s:35:\"asset::aws::archive\/images\/crux.jpg\";i:49;s:46:\"asset::aws::archive\/images\/dialogues_cover.jpg\";i:50;s:56:\"asset::aws::archive\/images\/law_of_one_book_one_cover.jpg\";i:51;s:41:\"asset::aws::archive\/images\/lightlines.jpg\";i:52;s:67:\"asset::aws::archive\/images\/living_the_law_of_one_101_1024_x_768.jpg\";i:53;s:68:\"asset::aws::archive\/images\/living_the_law_of_one_101_1600_x_1200.jpg\";i:54;s:66:\"asset::aws::archive\/images\/living_the_law_of_one_101_800_x_600.jpg\";i:55;s:62:\"asset::aws::archive\/images\/living_the_law_of_one_101_cover.jpg\";i:56;s:64:\"asset::aws::archive\/images\/living_the_law_of_one_101_desktop.jpg\";i:57;s:58:\"asset::aws::archive\/images\/living_the_law_of_one_cover.jpg\";i:58;s:34:\"asset::aws::archive\/images\/loo.jpg\";i:59;s:35:\"asset::aws::archive\/images\/loo1.jpg\";i:60;s:35:\"asset::aws::archive\/images\/loo2.png\";i:61;s:35:\"asset::aws::archive\/images\/loo3.png\";i:62;s:35:\"asset::aws::archive\/images\/loo4.png\";i:63;s:35:\"asset::aws::archive\/images\/loo5.png\";i:64;s:38:\"asset::aws::archive\/images\/mosaics.jpg\";i:65;s:37:\"asset::aws::archive\/images\/poetry.jpg\";i:66;s:67:\"asset::aws::archive\/images\/pub_light_lines_first_25_years_cover.jpg\";i:67;s:61:\"asset::aws::archive\/images\/pub_the_alphabet_mosaics_cover.jpg\";i:68;s:39:\"asset::aws::archive\/images\/quixotic.jpg\";i:69;s:33:\"asset::aws::archive\/images\/rc.png\";i:70;s:34:\"asset::aws::archive\/images\/rc1.png\";i:71;s:41:\"asset::aws::archive\/images\/rc1_german.png\";i:72;s:40:\"asset::aws::archive\/images\/rc1french.png\";i:73;s:34:\"asset::aws::archive\/images\/rc2.png\";i:74;s:41:\"asset::aws::archive\/images\/rc2_german.png\";i:75;s:42:\"asset::aws::archive\/images\/rc_homepage.png\";i:76;s:38:\"asset::aws::archive\/images\/secrets.jpg\";i:77;s:66:\"asset::aws::archive\/images\/the_ll_research_channeling_archives.jpg\";i:78;s:73:\"asset::aws::archive\/images\/the_poetry_of_carla_lisbeth_rueckert_cover.jpg\";i:79;s:38:\"asset::aws::archive\/images\/tilting.jpg\";i:80;s:44:\"asset::aws::archive\/images\/tilting_cover.jpg\";i:81;s:37:\"asset::aws::archive\/images\/voices.png\";i:82;s:37:\"asset::aws::archive\/images\/voices.psd\";i:83;s:43:\"asset::aws::archive\/images\/voices_cover.jpg\";i:84;s:41:\"asset::aws::archive\/images\/whatislove.jpg\";i:85;s:34:\"asset::aws::audio\/101\/lesson_1.mp3\";i:86;s:35:\"asset::aws::audio\/101\/lesson_10.mp3\";i:87;s:35:\"asset::aws::audio\/101\/lesson_11.mp3\";i:88;s:35:\"asset::aws::audio\/101\/lesson_12.mp3\";i:89;s:34:\"asset::aws::audio\/101\/lesson_2.mp3\";i:90;s:34:\"asset::aws::audio\/101\/lesson_3.mp3\";i:91;s:34:\"asset::aws::audio\/101\/lesson_4.mp3\";i:92;s:34:\"asset::aws::audio\/101\/lesson_5.mp3\";i:93;s:34:\"asset::aws::audio\/101\/lesson_6.mp3\";i:94;s:34:\"asset::aws::audio\/101\/lesson_7.mp3\";i:95;s:34:\"asset::aws::audio\/101\/lesson_8.mp3\";i:96;s:34:\"asset::aws::audio\/101\/lesson_9.mp3\";i:97;s:27:\"asset::aws::audio\/jenny.mp3\";i:98;s:44:\"asset::aws::audio\/meditations\/1979.07.11.mp3\";i:99;s:44:\"asset::aws::audio\/meditations\/1981.05.17.mp3\";}s:28:\"\u0000*\u0000escapeWhenCastingToString\";b:0;}}","batchId":null},"createdAt":1771917542,"delay":null}
 attempts = 0
 reserved_at =
 available_at = 1771917542
 created_at = 1771917542

But that's all I can figure out so far, I'm not sure what the problem is yet and I thought because it's costing so much I thought maybe someone else has some ideas on how to debug it or what to investigate more.

I do know there's an issue on Statamic v6 where I can't list my assets on S3 because it just times out? So maybe it's related too? I was able to make some progress on that one by importing the assets into eloquent on my production server, as we have a few thousand PDFs in there. But for the test website we've still been using the file driver instead of eloquent

It just seems to timeout with a 504 error on the assets page, so if it was something related to the queue and it was timing out too or getting a 504 error then maybe the job is being replayed over and over?

I have no clue... any other ideas or things to try?

How to reproduce

I wish I knew

Logs

Environment

Environment
Laravel Version: 12.52.0
PHP Version: 8.4.16
Composer Version: 2.9.5
Environment: testing
Debug Mode: ENABLED
Maintenance Mode: OFF
Timezone: America/Kentucky/Louisville
Locale: en

Cache
Config: NOT CACHED
Events: CACHED
Routes: CACHED
Views: CACHED

Drivers
Broadcasting: pusher
Cache: database
Database: app
Logs: stack / single
Mail: log
Queue: database
Session: database

Storage
public/storage: LINKED

Livewire
Livewire: v4.1.4

Statamic
Addons: 2
Sites: 28 (English, Arabic, Bulgarian, and 25 more)
Stache Watcher: Enabled
Static Caching: Disabled
Version: 6.3.3 PRO

Statamic Addons
mikomagni/statamic-environment: 6.x-dev
statamic/eloquent-driver: 5.2.0

Statamic Eloquent Driver
Addon Settings: file
Asset Containers: file
Assets: file
Blueprints: file
Collection Trees: file
Collections: file
Entries: eloquent
Fieldsets: file
Form Submissions: file
Forms: file
Global Sets: file
Global Variables: file
Navigation Trees: file
Navigations: file
Revisions: eloquent
Sites: file
Taxonomies: file
Terms: file
Tokens: file

Installation

Existing Laravel app

Additional details

I will let you know if I figure anything else out.