Build/CI improvements brainstorm tracker

We are seeing a lot of test flakes again which is a huge hindrance for developer productivity. This issue is for brainstorming various things that we might be able to build that would help us reduce CI time and reduce the number of reruns required for getting a successful test run. Once we do some brainstorming on high level ideas we can break out individual issues if we have people to work on them.

Some initial ideas to seed discussion:

• Have a UI that we can use to look at granular test data at the individual build/test level including test time and flake % (assuming we run with flake attempts > 1). The UI should allow for inspection of the overall history of test runs, and then allow drilling down and sorting so that we can look at the worst offenders and target them for fixing. Optimally, we would somehow also track when a test was added and last modified so that we can get an idea of whether a test has become more flaky over time. Google has a project called https://github.com/google/resultstoreui which we might be able to use for this, or we could also talk to EngFlow, BuildBuddy, etc. about potential SaaS solutions that they might be willing to work with us on as an OSS project.
• Increase flaky attempts to a larger number than we have now?
• Make sure flaky attempts are running on all builds that we do?
• Have some way of tracking the timeout for each test as well as how close we are to reaching it? This could be on the overall run as well as on a per test basis. This is something that could be surfaced in the above UI.
• Track most resource intensive tests in terms of RAM, CPU time, etc. so that we can make sure we are getting good bang for our testing buck. Look at trends over time.
• Have the UI able to report the worst offenders automatically via email, Slack plugin, GH issue, etc. so that we can make sure that things are getting triaged and dealt with.

cc @envoyproxy/envoy-maintainers @keith for further thoughts.