Multi-thread unpack and verify, unzipping slabs is the current bottleneck

Multi-thread unpack and verify. Unzipping slabs is the current bottleneck.

Summary

The current blk-archive unpack and verify operations appear to be bottlenecked by slab decompression, which is currently single-threaded. This limits performance on multi-core systems, especially when dealing with large archives or many slabs.

To improve throughput and scalability, we should parallelize the decompression of slabs and, where applicable, parallelize verification as well.

Goals

• Parallelize slab decompression to leverage available CPU cores
• Improve performance of both unpack and verify operations
• Maintain correctness and deterministic output
• Avoid unnecessary contention on shared state (e.g., output buffers)

Tasks

• Profile current slab decompression and identify single-threaded bottlenecks
• Investigate decompression strategy (e.g., per-slab, per-chunk parallelism)
• Refactor code to use a thread pool or parallel iterator (e.g., Rayon)
• Ensure output ordering is preserved where necessary (e.g., unpacked block order)
• Parallelize checksum/hash verification if done sequentially
• Add performance benchmarks before/after optimization
• Test on various archive sizes and core counts

Considerations

• Decompression must remain deterministic (important for verification)
• Memory usage may increase due to parallel buffers
• Output streams (e.g., writing to a block device) may require throttling or locking

Potential Benefits

• Significantly faster unpack/verify times
• Better utilization of modern multi-core systems
• Improved user experience for archive validation workflows