Improve Log Parsing and Loading Performance

Enhance the performance of the log ingestion pipeline, focusing on faster parsing and loading of log files, especially large or high-volume files.

Problem:

• Current log ingestion is slow or memory-intensive, leading to delays and potential timeouts or OutOfMemoryError exceptions.
• Performance bottlenecks may exist in file I/O, parsing logic, or database/storage writes.

Goals:

• Speed up log parsing and reduce memory consumption.
• Handle large files efficiently without crashing or hanging.
• Improve throughput for high-frequency log ingestion.

Proposed Improvements:

• 1. Replace Files.readAllBytes() / readString() with streaming methods (e.g., BufferedReader, Files.lines()).
• 2. Optimize parsing logic to avoid unnecessary object creation or string manipulation.
• 3. Use batch processing or async queues for loading parsed logs into storage.
• 4. Profile and monitor performance to identify slow points (e.g., with Java Flight Recorder or VisualVM).
• 5. Consider parallel processing for independent files or chunks of data.