Optimistic commit racing DropColumnFamily latches a fatal background error under the default parallel OCC validation

What

An optimistic transaction whose commit races a DropColumnFamily from another thread does not fail at conflict validation as #725 assumed. Under RocksDB's default OccValidationPolicy::kValidateParallel the commit is admitted past validation, fails inside the memtable inserter with Invalid argument: Invalid column family specified in write batch, and DBImpl::HandleMemTableInsertFailure latches that as a fatal background error on the whole environment. From that instant every write to every column family on the path fails with the same message until the database is closed and reopened (or, on 2.8.0, resume()d).

#725 fixed this for non-transactional putSync/removeSync (ignore_missing_column_families) and kept transactions out of scope on the premise that

Optimistic mode (the default, and the only mode Harper uses) already fails safely: conflict validation rejects a commit naming a dropped family early, with Invalid argument: Could not access column family <id>

That is only true of kValidateSerial. #726 tracks the same poisoning for pessimistic mode; this is the optimistic-mode counterpart, and it is the one Harper hits.

Why

OptimisticTransactionDB::Open(dbOptions, path, cfDescriptors, ...) (db_descriptor.cpp, the overload without OptimisticTransactionDBOptions) leaves validate_policy at its default, kValidateParallel. OptimisticTransaction::Commit() then takes CommitWithParallelValidate():

  Status s = TransactionUtil::CheckKeysForConflicts(db_impl, *tracked_locks_, true /* cache_only */);
  if (!s.ok()) return s;                       // "Could not access column family N" only if already dropped HERE
  s = db_impl->Write(write_options_, GetWriteBatch()->GetWriteBatch());   // plain Write, no callback

Validation runs before the commit enters the write thread. DBImpl::DropColumnFamilyImpl serializes only against write groups (write_thread_.EnterUnbatched around LogAndApply, which is where cfd->SetDropped() removes the family from the set), so a drop that starts after validation and holds the write thread through its MANIFEST write lands before the commit's Write() gets its turn. The batch still names the dropped family, MemTableInserter::SeekToColumnFamily fails it (the transaction's WriteOptions deliberately do not set ignore_missing_column_families), and HandleMemTableInsertFailure calls SetBGError(status, BackgroundErrorReason::kMemTable) — fatal severity, IsDBStopped() true, every later PreprocessWrite returns the stored status.

With kValidateSerial the validation is the write-group callback (WriteWithCallback), which runs inside the write thread, so the drop cannot interleave and the commit fails with the attributable Could not access column family N that #725 measured.

RocksDB's own LOG from a poisoned Harper run shows the two events 63 µs apart on two threads (main thread dropping, the commit lane committing):

[db/db_impl/db_impl.cc:3951] Dropped column family with id 3
[db/error_handler.cc:369] ErrorHandler: Set regular background error, auto_recovery=0, stop=1

Reproducer (binding only, rocksdb-js 2.7.1, poisons on the first round)

import { Worker, isMainThread, parentPort, workerData } from 'node:worker_threads';
import { RocksDatabase } from '@harperfast/rocksdb-js';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

if (isMainThread) {
	const path = mkdtempSync(join(tmpdir(), 'cf-race-'));
	const meta = RocksDatabase.open(path, { name: 'meta' });
	const worker = new Worker(new URL(import.meta.url), { workerData: { path } });
	worker.on('message', (m) => {
		if (m.type === 'ready') {
			const table = RocksDatabase.open(path, { name: 'table' });
			worker.postMessage('commit');
			setImmediate(() => table.dropSync()); // lands between the commit's validation and its Write()
		} else {
			console.log('commit error:', m.error);
			try { meta.putSync('probe', 1); console.log('env healthy'); }
			catch (e) { console.log('ENV POISONED ->', e.message); }
			worker.terminate(); meta.close();
		}
	});
} else {
	const table = RocksDatabase.open(workerData.path, { name: 'table' });
	parentPort.once('message', async () => {
		let error;
		try { await table.transaction((t) => t.putSync('k', Buffer.alloc(4096, 1))); } catch (e) { error = e.message; }
		parentPort.postMessage({ type: 'done', error });
	});
	parentPort.postMessage({ type: 'ready' });
}

commit error: Transaction commit failed: Invalid argument: Invalid column family specified in write batch
ENV POISONED -> Put failed: Invalid argument: Invalid column family specified in write batch

Note the commit's own error is the poison message, not Could not access column family — the write got past validation.

Impact

Harper drops a table's column families from the thread that runs drop_table while other worker threads can still be committing to that table (a source-fill cache write that resolved its caller before committing, an eviction batch, a plain put admitted before the schema change reached that worker). One such commit turns every Harper database on that path read-only until restart: the drop's own catalog cleanup fails and leaves the dropping tombstone, every worker's interrupted-drop completion fails on the latched error, and a same-name create cannot proceed. Tracked on the Harper side as harper#1381 (public CI runs 33076221779 and 33139269321 on main, integrationTests/apiTests/blob.test.mjs). Affects 2.7.1 (what Harper main pins) and 2.8.0 — the open path is unchanged there; 2.8.0's getLastError()/resume() only adds observability and recovery after the fact.

Fix directions

1. Serialize drops against in-flight optimistic commits in the binding — the interlock #726 describes (per-column-family in-flight touch count taken on transactional Put/Delete, released on commit/rollback; Database::Drop/DropSync wait for it to reach zero, coordinated with columnsMutex and the CommitWorker lane). Fixes both modes with no change to the commit path's parallelism.
2. Open with OptimisticTransactionDBOptions{ validate_policy = kValidateSerial } — a one-line change that moves validation into the write thread, restoring the behaviour #725 assumed. Costs commit parallelism under contention (validation joins the leader's critical section), which is why RocksDB made parallel the default.
3. ignore_missing_column_families on transaction write options is not an option: #725 measured it turning a commit that spans a live and a dropped family into a silent partial commit.

Either 1 or 2 makes a drop/commit race a contained, attributable failure of that one commit instead of a node-wide write outage.

---

Filed by Claude Fable 5 while root-causing harper#1381.