Timing data should sometimes be saved for failed builds

Open
#244 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
performance, testing

Research direction

No files or tests are named. Trace where saved timing data is updated after builds, then inspect how failed atoms and the age of the last successful timing data are represented. Done means failed builds can provide timing data when none exists or when successful data is sufficiently old, without replacing fresh successful data unnecessarily.

Written by the indexing model from the issue text.

Description

Currently we only update saved timing data for a job when a build is successful (no failed atoms). This makes sense in general -- failed builds might have unrepresentative timing data.

I think there are situations though when we might want to update timing data even when atoms fail. The situations I was thinking of were

  • when no timing data currently exists. When a job config is first created, no timing data exists and it's common for users to still be working out the kinks in their config, so there may be many initial builds that all fail. It would be better if clusterrunner would save timing data even for failed builds if it doesn't have data for a successful build.
  • when a build has not passed in a long time. We've seen in practice that old timing data can actually have a worse affect on build times than having no timing data at all. If there is something weird about a job causing builds to fail and nobody notices, this can cause build times to slowly increase over time as the timing data gets more and more outdated.
Dominant language
Python
Stars
184
Forks
41
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from box/ClusterRunner

All issues in box/ClusterRunner

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.