sphinx-doc/sphinx

Cannot recover from MemoryErrors

Chiusa

#7635 aperta il 8 mag 2020

Vedi su GitHubGuida contributor
 (2 commenti) (0 reazioni) (0 assegnatari)Python (1985 fork)batch import
help wantedtype:bug

Metriche repository

Star
 (5625 stelle)
Metriche merge PR
 (Merge medio 10g 11h) (11 PR mergiate in 30 g)

Descrizione

We are using recommonmark with sphinx to parse 2.3k+ md files and experiencing excesive memory usage. Running 64 bit we see a peak of about 19gb of memory usage and an execution time around 30 minutes. On many machines (or running 32 bit) this is not doable. The majority of the markdown files have an rst label, and some of them have references to these labels. Are there configuration options that may be affecting this? Is there a way to review what's actually taking up this memory?

Guida contributor

Direzione di ricerca
Investigate memory usage during parsing of many markdown files with recommonmark. Use memory profiler or tracemalloc to identify peak memory allocations. Compare with direct Python memory usage and explore Sphinx's handling of large document sets. Look at recommonmark's parser and Sphinx's build process in sphinx/builders/ for potential optimizations or configuration options to reduce memory footprint.
Tech stack
python
Dominio
documentationperformance
Tipo issue
Bug
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
4
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
Oltre 1 settimana
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
Datata
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
Richiede indagine
Prerequisiti
PythonMemory profilingSphinx internalsCPython debugging
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
10