project-chip/connectedhomeip

Parallelize code generation for jinja files and code formatting

Open

#38,775 opened on May 6, 2025

View on GitHub
 (1 comment) (0 reactions) (0 assignees)C++ (6,410 stars) (1,659 forks)batch import
good first issuestale

Description

We want to have fast iteration speed on code generation. https://github.com/project-chip/connectedhomeip/blob/master/scripts/py_matter_idl/matter/idl/generators/cpp/sdk/sdk_generator.py#L120 currently generates a LOT of files and could benefit from paralelization

Update code logic to do parallel generation and test out if this improves runtime (i.e. if it is worth it - this issue was created on the assumption that it is, however we should verify that).

We also do file formatting and those seem to call one single exec to format many files at once. We should investigate if splitting files per cpu count improves formatting (i.e. is clang-format and kotlin-format single threaded or multi-threaded?)

Contributor guide

Tech stack
pythoncppkotlin
Domain
toolingbuild system
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PythonConcurrency basicsBuild system
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45
Research direction
Investigate the file scripts/py matter idl/matter/idl/generators/cpp/sdk/sdk generator.py at line 120 to understand the current sequential generation. Implement parallel generation using Python's concurrent.futures module. Measure runtime before and after to confirm improvement. Additionally, examine clang format and kotlin format to determine if they are single threaded; if so, split the file list by CPU count and format in parallel. Report the performance results in the issue.