Investigate and reduce memory and CPU usage on large code bases

Open
  • Is something wrong?
#119 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
php
Domain
performance

Research direction

Start with phpbench.json and benchmarks/, then inspect collectFromPaths() and the Parser visitors named in the issue. Add a repeatable benchmark with run instructions, and measure memory and time on a pinned large code base before investigating the listed hypotheses. Done means the benchmark and a write-up are in the repo, with any worthwhile improvements supported by before and after measurements.

Written by the indexing model from the issue text.

Description

enhancement

Background

#91 found that the batching and streaming report API never reduced memory, so it is being deprecated in #118.
Memory on large code bases is still a real concern, though.
This ticket investigates where the memory and time actually go, and improves them where the measurements justify it.

Current numbers

Measured with bin/phpcca analyse <path> --report-type=json in the project's Docker image (PHP 8.5), cache disabled:

Code base Files Classes Peak memory
src/ 150 – 18 MB
vendor/ 5,653 4,499 164 MB

Around 5,000 files already exceeds PHP's common CI default of memory_limit=128M.

Hypotheses to check (measure first)

  1. Unused metrics are always calculated and kept.
    Parser always runs the Halstead and cyclomatic visitors.
    Every CognitiveMetrics keeps a HalsteadMetrics, CyclomaticMetrics and UnderstandabilityMetrics object, plus the cyclomatic decision-point breakdown, even when showHalsteadComplexity, showCyclomaticComplexity and showUnderstandability are off.
    Skipping or dropping them when they are not shown, sorted on, or used in a report could save memory and CPU.
  2. Per-method object size.
    CognitiveMetrics has about 40 properties, including 12 nullable Delta objects that are only used with a baseline.
    Check the real size per method and whether lazy or compact storage is worth it.
  3. Visitor state across files.
    Check that per-file visitor state (for example Halstead class and method metrics, and the annotation visitor's ignored items) is released after each file and does not grow with the project.
  4. File list materialisation.
    collectFromPaths() turns the whole file iterator into an array.
    Probably small, but confirm.
  5. Cache hits.
    Compare memory and time for a cold run, a warm-cache run, and a run with the cache disabled.

Approach

  1. Add a repeatable benchmark: phpbench is already configured (phpbench.json, benchmarks/), or a small script that reports memory_get_peak_usage(true) and wall time.
    Use a large, pinned code base such as vendor/ or a fixed Symfony checkout.
  2. Profile memory (for example with memory_get_usage() checkpoints, or php-meminfo or Blackfire) to find the biggest contributors before changing code.
  3. Implement only the changes that show a clear win, one per PR, with before and after numbers in the PR description.
  4. Avoid behaviour changes.
    Output, reports, baselines and cache files must stay the same.
    If a change needs a public API break, deprecate first and remove in 2.0, like #91 and #92.

Done when

  • The benchmark is in the repo, with instructions to run it.
  • This issue has a short write-up of where memory and time go.
  • Every improvement that clearly pays off is merged with measured numbers, or this issue documents why none was worth it.
Dominant language
PHP
Stars
55
Forks
3
Avg merge
25m
Merged PRs (30d)
22

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Phauthentic/cognitive-code-analysis

All issues in Phauthentic/cognitive-code-analysis

Similar issues

More PHP issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.