Migration v343 logs multiple MongoDB IndexNotFound errors during upgrade from 8.9.1 to 9.0.0 at the first startup

The errors come from the drop-index calls in server/startup/migrations/v343.ts, which run through the migration runner in server/lib/migrations.ts. Read v343.ts first, then check how Promise.allSettled handles the rejected IndexNotFound (code 27) results. Done means a missing index no longer produces an error stack trace, while real failures still surface. Open PR #43712 already targets this, so review it before starting new work.

Migration v343 logs multiple MongoDB IndexNotFound errors during upgrade from 8.9.1 to 9.0.0

Description

After upgrading Rocket.Chat from 8.9.1 to 9.0.0, multiple MongoServerError: IndexNotFound errors appeared during the first startup while database migration v343 was running.

The migration attempted to drop several MongoDB indexes that did not exist in the database.

Despite these errors, all migrations completed successfully, and Rocket.Chat 9.0.0 started normally.

I would like to report this behavior to clarify whether these errors are expected during the upgrade process or whether migration v343 should handle missing indexes more gracefully.

Environment

Component	Version
Previous Rocket.Chat	8.9.1
New Rocket.Chat	9.0.0
MongoDB	8.2.3
Node.js	24.22.1
OpenSSL	3.5.9
Deployment	Docker Compose
Platform	Linux x64
MongoDB Storage Engine	WiredTiger
Database Migration	335 → 353

Steps to Reproduce

1. Start with an existing Rocket.Chat 8.9.1 installation.

2. Upgrade the Rocket.Chat Docker image to version 9.0.0.

3. Start Rocket.Chat using Docker Compose.

4. Monitor the startup logs using:

   docker compose logs -f rocketchat

5. Observe the errors during migration v343.

Actual Behavior

During the first startup, Rocket.Chat begins migrating the database:

Migrating between versions
fromVersion: 335
toVersion: 353

When migration v343 is executed, multiple errors are logged:

Running migration
direction: up
version: 343

Error dropping index MongoServerError:
index not found with name [ts_1_reports.ts_1]

code: 27
codeName: IndexNotFound

The following indexes were reported as missing:

ts_1_reports.ts_1
message.u._id_1_ts_1
reportedUser._id_1_ts_1
message.rid_1_ts_1
message._id_1_ts_1
userId_1_ts_1

The stack traces point to:

server/startup/migrations/v343.ts:11:18
server/lib/migrations.ts:156:2
server/lib/migrations.ts:276:6
server/lib/migrations.ts:295:4
server/startup/migrations/xrun.ts:96:2

However, the migration process continues and eventually reports:

Running migration
direction: up
version: 353

Finished migrating.

Rocket.Chat subsequently starts successfully:

SERVER RUNNING

Rocket.Chat Version: 9.0.0
NodeJS Version: 24.22.1
MongoDB Version: 8.2.3
MongoDB Engine: wiredTiger

The MongoDB connection and internal services also initialize successfully.

Expected Behavior

If an index targeted for removal does not exist, the migration should ideally handle this situation without logging a full error stack trace.

Possible improvements:

• Check whether an index exists before attempting to drop it.
• Gracefully handle MongoDB error code 27 (IndexNotFound).
• Log missing indexes as informational messages or warnings instead of errors when their absence is harmless.

Impact

• Multiple error stack traces during the first startup after upgrading.
• No observed interruption of the database migration.
• Rocket.Chat 9.0.0 starts successfully.
• No data loss was observed in the provided startup logs, although data integrity was not independently verified.

The messages may unnecessarily alarm administrators performing production upgrades.

Questions

1. Is this behavior expected when upgrading from Rocket.Chat 8.9.1 to 9.0.0?
2. Is migration v343 intentionally designed to tolerate missing indexes?
3. Could missing indexes be handled without producing full error stack traces?
4. Is any additional database verification recommended after encountering these errors?

Additional Information

The errors occurred during the first startup immediately after upgrading to Rocket.Chat 9.0.0.

All subsequent migration steps completed, and the server reached the SERVER RUNNING state.

Thank you for investigating.

Append:
Best greeting to the Databaes Developer from an DBA :-)

rocketchat | Error dropping index MongoServerError: index not found with name [ts_1_reports.ts_1]
rocketchat | at Connection.sendCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:299:27)
rocketchat | at processTicksAndRejections (node:internal/process/task_queues:104:5)
rocketchat | at Connection.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:327:26)
rocketchat | at Server.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:168:29)
rocketchat | at DropIndexOperation.executeCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/command.js:76:16)
rocketchat | at DropIndexOperation.execute (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/indexes.js:141:16)
rocketchat | at tryOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:207:20)
rocketchat | at executeOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:75:16)
rocketchat | at Collection.dropIndex (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:373:16)
rocketchat | at async Promise.allSettled (index 6)
rocketchat | at Object.up (server/startup/migrations/v343.ts:11:18)
rocketchat | at migrate (server/lib/migrations.ts:156:2)
rocketchat | at server/lib/migrations.ts:276:6
rocketchat | at migrateDatabase (server/lib/migrations.ts:295:4)
rocketchat | at performMigrationProcedure (server/startup/migrations/xrun.ts:96:2)
rocketchat | at startup (server/startup/index.ts:17:2) {
rocketchat | errorLabelSet: Set(0) {},
rocketchat | errorResponse: {
rocketchat | ok: 0,
rocketchat | errmsg: 'index not found with name [ts_1_reports.ts_1]',
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: [Object]
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | },
rocketchat | ok: 0,
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: {
rocketchat | hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
rocketchat | keyId: 0
rocketchat | }
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | }
rocketchat | Error dropping index MongoServerError: index not found with name [message.u._id_1_ts_1]
rocketchat | at Connection.sendCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:299:27)
rocketchat | at processTicksAndRejections (node:internal/process/task_queues:104:5)
rocketchat | at Connection.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:327:26)
rocketchat | at Server.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:168:29)
rocketchat | at DropIndexOperation.executeCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/command.js:76:16)
rocketchat | at DropIndexOperation.execute (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/indexes.js:141:16)
rocketchat | at tryOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:207:20)
rocketchat | at executeOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:75:16)
rocketchat | at Collection.dropIndex (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:373:16)
rocketchat | at async Promise.allSettled (index 7)
rocketchat | at Object.up (server/startup/migrations/v343.ts:11:18)
rocketchat | at migrate (server/lib/migrations.ts:156:2)
rocketchat | at server/lib/migrations.ts:276:6
rocketchat | at migrateDatabase (server/lib/migrations.ts:295:4)
rocketchat | at performMigrationProcedure (server/startup/migrations/xrun.ts:96:2)
rocketchat | at startup (server/startup/index.ts:17:2) {
rocketchat | errorLabelSet: Set(0) {},
rocketchat | errorResponse: {
rocketchat | ok: 0,
rocketchat | errmsg: 'index not found with name [message.u._id_1_ts_1]',
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: [Object]
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | },
rocketchat | ok: 0,
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: {
rocketchat | hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
rocketchat | keyId: 0
rocketchat | }
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | }
rocketchat | Error dropping index MongoServerError: index not found with name [reportedUser._id_1_ts_1]
rocketchat | at Connection.sendCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:299:27)
rocketchat | at processTicksAndRejections (node:internal/process/task_queues:104:5)
rocketchat | at Connection.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:327:26)
rocketchat | at Server.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:168:29)
rocketchat | at DropIndexOperation.executeCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/command.js:76:16)
rocketchat | at DropIndexOperation.execute (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/indexes.js:141:16)
rocketchat | at tryOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:207:20)
rocketchat | at executeOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:75:16)
rocketchat | at Collection.dropIndex (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:373:16)
rocketchat | at async Promise.allSettled (index 8)
rocketchat | at Object.up (server/startup/migrations/v343.ts:11:18)
rocketchat | at migrate (server/lib/migrations.ts:156:2)
rocketchat | at server/lib/migrations.ts:276:6
rocketchat | at migrateDatabase (server/lib/migrations.ts:295:4)
rocketchat | at performMigrationProcedure (server/startup/migrations/xrun.ts:96:2)
rocketchat | at startup (server/startup/index.ts:17:2) {
rocketchat | errorLabelSet: Set(0) {},
rocketchat | errorResponse: {
rocketchat | ok: 0,
rocketchat | errmsg: 'index not found with name [reportedUser._id_1_ts_1]',
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: [Object]
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | },
rocketchat | ok: 0,
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: {
rocketchat | hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
rocketchat | keyId: 0
rocketchat | }
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | }
rocketchat | Error dropping index MongoServerError: index not found with name [message.rid_1_ts_1]
rocketchat | at Connection.sendCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:299:27)
rocketchat | at processTicksAndRejections (node:internal/process/task_queues:104:5)
rocketchat | at Connection.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:327:26)
rocketchat | at Server.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:168:29)
rocketchat | at DropIndexOperation.executeCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/command.js:76:16)
rocketchat | at DropIndexOperation.execute (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/indexes.js:141:16)
rocketchat | at tryOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:207:20)
rocketchat | at executeOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:75:16)
rocketchat | at Collection.dropIndex (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:373:16)
rocketchat | at async Promise.allSettled (index 9)
rocketchat | at Object.up (server/startup/migrations/v343.ts:11:18)
rocketchat | at migrate (server/lib/migrations.ts:156:2)
rocketchat | at server/lib/migrations.ts:276:6
rocketchat | at migrateDatabase (server/lib/migrations.ts:295:4)
rocketchat | at performMigrationProcedure (server/startup/migrations/xrun.ts:96:2)
rocketchat | at startup (server/startup/index.ts:17:2) {
rocketchat | errorLabelSet: Set(0) {},
rocketchat | errorResponse: {
rocketchat | ok: 0,
rocketchat | errmsg: 'index not found with name [message.rid_1_ts_1]',
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: [Object]
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | },
rocketchat | ok: 0,
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: {
rocketchat | hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
rocketchat | keyId: 0
rocketchat | }
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | }
rocketchat | Error dropping index MongoServerError: index not found with name [message._id_1_ts_1]
rocketchat | at Connection.sendCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:299:27)
rocketchat | at processTicksAndRejections (node:internal/process/task_queues:104:5)
rocketchat | at Connection.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:327:26)
rocketchat | at Server.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:168:29)
rocketchat | at DropIndexOperation.executeCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/command.js:76:16)
rocketchat | at DropIndexOperation.execute (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/indexes.js:141:16)
rocketchat | at tryOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:207:20)
rocketchat | at executeOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:75:16)
rocketchat | at Collection.dropIndex (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:373:16)
rocketchat | at async Promise.allSettled (index 10)
rocketchat | at Object.up (server/startup/migrations/v343.ts:11:18)
rocketchat | at migrate (server/lib/migrations.ts:156:2)
rocketchat | at server/lib/migrations.ts:276:6
rocketchat | at migrateDatabase (server/lib/migrations.ts:295:4)
rocketchat | at performMigrationProcedure (server/startup/migrations/xrun.ts:96:2)
rocketchat | at startup (server/startup/index.ts:17:2) {
rocketchat | errorLabelSet: Set(0) {},
rocketchat | errorResponse: {
rocketchat | ok: 0,
rocketchat | errmsg: 'index not found with name [message._id_1_ts_1]',
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: [Object]
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | },
rocketchat | ok: 0,
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: {
rocketchat | hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
rocketchat | keyId: 0
rocketchat | }
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | }
rocketchat | Error dropping index MongoServerError: index not found with name [userId_1_ts_1]
rocketchat | at Connection.sendCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:299:27)
rocketchat | at processTicksAndRejections (node:internal/process/task_queues:104:5)
rocketchat | at Connection.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:327:26)
rocketchat | at Server.command (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:168:29)
rocketchat | at DropIndexOperation.executeCommand (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/command.js:76:16)
rocketchat | at DropIndexOperation.execute (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/indexes.js:141:16)
rocketchat | at tryOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:207:20)
rocketchat | at executeOperation (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/execute_operation.js:75:16)
rocketchat | at Collection.dropIndex (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:373:16)
rocketchat | at async Promise.allSettled (index 11)
rocketchat | at Object.up (server/startup/migrations/v343.ts:11:18)
rocketchat | at migrate (server/lib/migrations.ts:156:2)
rocketchat | at server/lib/migrations.ts:276:6
rocketchat | at migrateDatabase (server/lib/migrations.ts:295:4)
rocketchat | at performMigrationProcedure (server/startup/migrations/xrun.ts:96:2)
rocketchat | at startup (server/startup/index.ts:17:2) {
rocketchat | errorLabelSet: Set(0) {},
rocketchat | errorResponse: {
rocketchat | ok: 0,
rocketchat | errmsg: 'index not found with name [userId_1_ts_1]',
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: [Object]
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | },
rocketchat | ok: 0,
rocketchat | code: 27,
rocketchat | codeName: 'IndexNotFound',
rocketchat | '$clusterTime': {
rocketchat | clusterTime: new Timestamp({ t: 1794555230, i: 80 }),
rocketchat | signature: {
rocketchat | hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
rocketchat | keyId: 0
rocketchat | }
rocketchat | },
rocketchat | operationTime: new Timestamp({ t: 1794555230, i: 80 })
rocketchat | }

But:
rocketchat | +-----------------------------------------------------------------------------------------+
rocketchat | | SERVER RUNNING |
rocketchat | +-----------------------------------------------------------------------------------------+
rocketchat | | |
rocketchat | | Rocket.Chat Version: 9.0.0 |
rocketchat | | NodeJS Version: 24.22.1 - x64 |
rocketchat | | MongoDB Version: 8.2.3 |
rocketchat | | MongoDB Engine: wiredTiger |
rocketchat | | Platform: linux |
rocketchat | | Process Port: 3000 |
rocketchat | | Site URL: https://rocketchat.deutschland.de |
rocketchat | | Hashed Site URL: 1a918a270a065a848882a9443180d1dd7a2afe26e091d006ef64583f5480adba |
rocketchat | | OpenSSL Version: 3.5.9 |
rocketchat | | FIPS Provider: Disabled |
rocketchat | | Commit Hash: 756135f4a9 |
rocketchat | | Commit Branch: HEAD |
rocketchat | | |
rocketchat | +-----------------------------------------------------------------------------------------+