Cached failed action is used when retrying a failed build

Open
#2,020 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
build-system, cli

Research direction

Start by tracing the --retry-failed command-line path and the action-cache lookup used during remote execution. Reproduce the logged scenario with a failed cached action; done means retrying bypasses the cached failure and runs the build action again.

Written by the indexing model from the issue text.

Description

bug remote execution

When using remote execution, if a build fails the failure is saved in the action cache. However builds can fail for a variety of reasons (e.g. running out of resources) that are not necessarily reproducible.

Buildstream offers to retry at the command line and also offers the --retry-failed command line argument for this purpose. However, it will still get a cached failure from the action cache.

Here is an example log (I extracted just the things relevant to the element in question):

[11:08:35][--:--:--][8524a2f2][    pull:freedesktop-sdk.bst:components/fontconfig.bst] START   freedesktop-sdk/components-fontconfig/8524a2f2-pull.20250611-110835.log
[11:08:35][--:--:--][8524a2f2][    pull:freedesktop-sdk.bst:components/fontconfig.bst] STATUS  Pulling artifact 8524a2f2 <- https://gbm.gnome.org:11004
[11:08:35][--:--:--][8524a2f2][    pull:freedesktop-sdk.bst:components/fontconfig.bst] STATUS  Pulling data for artifact 8524a2f2 <- https://gbm.gnome.org:11004
[11:08:35][--:--:--][8524a2f2][    pull:freedesktop-sdk.bst:components/fontconfig.bst] INFO    Pulled artifact 8524a2f2 <- https://gbm.gnome.org:11004
[11:08:35][--:--:--][8524a2f2][    pull:freedesktop-sdk.bst:components/fontconfig.bst] INFO    Discarded failed build
[11:08:35][00:00:00][8524a2f2][    pull:freedesktop-sdk.bst:components/fontconfig.bst] SKIPPED Pull
[11:08:35][--:--:--][8524a2f2][   fetch:freedesktop-sdk.bst:components/fontconfig.bst] START   freedesktop-sdk/components-fontconfig/8524a2f2-fetch.20250611-110835.log
[11:08:35][--:--:--][8524a2f2][   fetch:freedesktop-sdk.bst:components/fontconfig.bst] STATUS  Pulling source 031032b8 <- https://gbm.gnome.org:11004
[11:08:36][--:--:--][8524a2f2][   fetch:freedesktop-sdk.bst:components/fontconfig.bst] STATUS  Pulling data for source 031032b8 <- https://gbm.gnome.org:11004
[11:08:36][--:--:--][8524a2f2][   fetch:freedesktop-sdk.bst:components/fontconfig.bst] INFO    Pulled source 031032b8 <- https://gbm.gnome.org:11004
[11:08:36][00:00:00][8524a2f2][   fetch:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS freedesktop-sdk/components-fontconfig/8524a2f2-fetch.20250611-110835.log
[11:08:37][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] START   freedesktop-sdk/components-fontconfig/8524a2f2-build.20250611-110837.log
[11:08:37][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] START   Staging dependencies at: /
[11:09:04][00:00:26][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Staging dependencies at: /
[11:09:04][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] START   Integrating sandbox
[11:09:32][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] INFO    Action result found in action cache
[11:09:48][00:00:44][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Integrating sandbox
[11:09:48][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] START   Staging sources
[11:09:48][00:00:00][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Staging sources
[11:09:48][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] START   Running commands
[11:09:48][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] INFO    Action result found in action cache
[11:09:49][00:00:00][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] FAILURE Running commands
[11:09:49][--:--:--][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] START   Caching artifact
[11:09:49][00:00:00][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Caching artifact
[11:09:49][00:01:11][8524a2f2][   build:freedesktop-sdk.bst:components/fontconfig.bst] FAILURE Command failed

You can see that bst pulled a cached failure and promptly discarded it, but it then proceeded to get the failed action from the action cache and not actually retry the build.

Dominant language
Python
Stars
144
Forks
45
Avg merge
20h 38m
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/buildstream

All issues in apache/buildstream

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.