Documentation Request: Project Flow and Key Functions

Problem

The codebase lacks documentation explaining:

• How data flows through Bronze → Silver → Gold layers
• What key functions do and when they're called
• How the pipeline orchestrates everything

Requested Documentation

1. Architecture Overview (docs/architecture.md)

• Data flow diagram showing the 3 layers
• How each layer transforms data
• Database schema relationships

2. Key Functions Reference (docs/functions.md)

Bronze Layer (src/raw/raw.service.ts)

• ingestEachUserInTheirRepos() (line 868) - Main entry point, discovers repos per user
• discoverReposForUser() (line 543) - Finds repos with user activity
• buildRepoUsersMap() (line 592) - Creates repo → users mapping
• ingestIssuesAndPRsByCreator() (line 607) - Fetches issues/PRs
• ingestIssueComments() (line 670) - Fetches issue comments with parent ID resolution
• ingestPRReviewComments() (line 711) - Fetches PR review comments
• ingestCommitsForUsers() (line 752) - Fetches commits per user
• ingestCommitsForPR() (line 514) - Fetches commits for specific PR
• retryWithBackoff() (line 37) - Handles rate limits and server errors

Silver Layer (src/normalized/)

• SilverOrchestratorService.buildBundle() (line 26) - Coordinates normalization
• Each service's get*Since() methods - Load, map, deduplicate Bronze data
• Mappers in mappers.ts - Transform Bronze to normalized format

Gold Layer (src/analytics/)

• AnalyticsService.refreshAll() (line 26) - Transforms Silver → Gold
• mapSilverToCurated() - Aggregates activity data
• AnalyticsReportService.generateFrontendReport() (line 37) - Creates user reports

Pipeline (src/pipeline/pipeline.service.ts)

• run() (line 54) - Orchestrates Bronze → Silver → Gold → Report
• addNewUsers() (line 117) - Async processing for new users (6 months data)
• removeUsers() (line 92) - Cascade deletion across all layers

3. Time Windows Strategy (docs/time-windows.md)

• New users: 180 days of history
• Existing users: 48 hours incremental updates
• How userTimeWindows map works (line 901-911)

4. Error Handling (docs/error-handling.md)

• Rate limit recovery with exponential backoff
• Server error (5xx) handling
• Missing data scenarios

5. Parent ID Resolution (docs/parent-resolution.md)

• Why comments need parent IDs
• resolveIssueParentId() (line 469) and resolvePRParentId() (line 489)
• Number → ID mapping with buildNumberToIdMap() (line 454)

Deliverables

• Architecture diagram (Mermaid or image)
• Function reference with examples
• Code snippets showing typical flows
• Time windows explanation

Priority

High - Reduces onboarding time for new contributors

Labels

documentation, help-wanted