envoyproxy/envoy

Set up proper caching for mobile CI

Open

#26,970 opened on Apr 26, 2023

View on GitHub
 (6 comments) (0 reactions) (0 assignees)C++ (5,373 forks)batch import
area/ciarea/test flakesenhancementhelp wantedmobileno stalebot

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

Recently, we have added a bunch of caching to Envoy's CI which has both sped things up and fixed (or seriously mittigated) a load of persistent flakes that have been endemic (optimistically i would say until ~now)

Essentially there are a few things worth caching:

  • docker build image
  • bazel directories (repository_cache, external, .cache)

Regarding the docker image it is huge - to make it work faster than pulling from dockerhub in Envoy's CI i have had to load it using tmpfs - any other way would be significantly slower

I held off doing this for a long time because dockerhub can be faster - but now that we have done this it has been a huge improvement to both performance and reliability. Anecdotally i would say that the azp cache gets faster the more you use it whereas the reverse seems to be true with the dockerhub cache (with some caveats)

Likewise with the bazel directories, a huge number of flakes occur just because of transient network issues, and caching these mostly eliminates this problem

While working on this for Envoy one thing i noticed was that incorrect use of bazel was creating multiple environments, fixing that was a small but sure optimization. Envoy now dumps pretty good fs info at the end of the build, and this is invaluable to understanding what bazel is doing and why it goes wrong - i strongly suggest we do the same for the mobile ci

Contributor guide