pyodide/pyodide

Optimize the size of minimal pyodide build

Offen

#646 geöffnet am 09.05.2020

 (44 Kommentare) (24 Reaktionen) (0 zugewiesene Personen)Python (735 Forks)batch import
buildhelp wantedperformancesize & load time optimization

Repository-Metriken

Stars
 (10.883 Sterne)
PR-Merge-Metriken
 (Durchschn. Merge 1T 20h) (48 gemergte PRs in 30 T)

Beschreibung

I have been reading https://pythontips.com/2019/05/22/running-python-in-the-browser/ which does a nice comparison on tools to run python in the browser. The main downside mentioned for pyodide there was the size of downloads. While optimizing the size of numpy, pandas etc is probably more challenging, I think we can reduce the size of downloads in the case of a minimal pyodide build with just cpython, pyodide tools and micropip.

I would propose to introduce en optional env variable PYODIDE_MINIMAL=true that together with partial builds in https://github.com/iodide-project/pyodide/pull/645 could be used as follows,

PYODIDE_MINIMAL=true PYODIDE_PACKAGES="micropip" make

This would,

  • build with USE_FREETYPE=0 (it is statically linked as far as I understand? libfreetype.a is 1.1MB), USE_PNG=0 (libpng.a is 350k) which were initially needed for matplotlib I think. Done in #658
  • remove some of the left over test folders in stdlib,
    Lib/distutils/tests
    Lib/sqlite3/test
    Lib/unittest/test
    
    that are now up to 800 kB in size. The main Lib/test (30MB) is already packaged as a separate python module. for these I think we can probably just remove them when PYODIDE_MINIMAL=true and keep running these tests in CI otherwise.
  • re-evaluate if some of the other larger and exotic stdlib modules can be removed with remove_modules.txt. For instance pydoc and pydoc_data are around 1MB in size. There are some .exe files in Lib/distutils/command/ for ~2MB (not sure if they are actually included in pyodide). Edit: installers removed in https://github.com/iodide-project/pyodide/pull/647
  • re-evaluate if some of the modules build in the main Makefile and included by default are still useful in this use-case. For instance, jeudi + parso is 6MB. Done in #658
  • on the contrary I would have included micropip by default instead of a making it a standalone package. the issue is that the distlib dependency is 500k.
  • generally list all files in the mounted folder from within pyodide and see if anything is not relevant there.

I'm not sure how these sizes would map to the size of compressed pyodide package, but I'm hoping we can improve the current situation which is,

-rw-r--r-- 1 circleci circleci  8.1M May  8 21:35 pyodide.asm.data
-rw-r--r-- 1 circleci circleci  328K May  8 21:35 pyodide.asm.data.js
-rw-r--r-- 1 circleci circleci  3.2M May  8 21:35 pyodide.asm.js
-rw-r--r-- 1 circleci circleci   13M May  8 21:35 pyodide.asm.wasm
-rw-r--r-- 1 circleci circleci   16K May  8 21:35 pyodide.js

Total: 24.6 MB

(package size are tracked in the "check-size" step of the "build" CI job).

Some of this could also benefit the full pyodide build, not just the minimal version.

@phorward would you be interested in investigating some of it? We probably need https://github.com/iodide-project/pyodide/pull/645 merged first.

Contributor Guide