Investigate/Optimize the Content Defined Chunking (CDC) used to improve performance

Investigate/Optimize the Content Defined Chunking (CDC) used to improve performance

Summary

The current implementation of Content Defined Chunking (CDC) in blk-archive may benefit from performance improvements, especially when handling large volumes of data or high-throughput use cases. This issue is to investigate the existing CDC implementation and explore opportunities for optimization.

Goals

• Evaluate the performance characteristics of the current CDC algorithm
• Identify bottlenecks in throughput, CPU usage, memory, or hashing
• Benchmark with real-world data sets to assess practical performance
• Explore faster alternatives or parameter tuning (e.g., rolling hash window size, minimum/average/max chunk size)
• Consider SIMD or multi-threaded enhancements if applicable

Tasks

• Profile CDC during blk-archive pack and identify hot spots
• Evaluate the impact of chunk size thresholds on de-duplication and performance
• Compare existing implementation against other CDC algorithms
• Investigate whether CDC processing can be parallelized or batched efficiently or utilize GPUs
• Assess cache and memory efficiency of current chunk boundary detection
• Document trade-offs between de-duplication effectiveness and processing speed

References

• FastCDC: Fast and Efficient Content-Defined Chunking
• Rabin fingerprinting

Considerations

• Maintain compatibility with existing slab/archive formats (or version accordingly)
• Balance performance improvements with de-duplication effectiveness

Potential Benefits

• Faster archive pack times
• Reduced CPU usage in high-volume workflows
• Improved scalability on multi-core/GPU enabled systems