Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Inconsistent macOS unit tests failures in CI

Open
  • Is something wrong?
#217 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
swift
Domain
ci-cd, testing

Research direction

Start by inspecting the Buildkite macOS test step and the failed and passed builds linked in the issue; no repository file or test entry point is named. Compare agent and signing details to isolate the intermittent cancellation, then verify that the same commit completes the tests consistently across agents.

Written by the indexing model from the issue text.

Description

You may have noticed some builds failing the macOS tests step. Emphasis on some, because this doesn't seem to be related to the code but to the agent running the builds.

The failure
[16:45:44]: ▸ Build Succeeded
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksiOS_AutomatticTracksModelObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticRemoteLogging' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksConstantsObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksModelObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksiOS_AutomatticTracksTests' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticEncryptedLogs' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksEventsForSwift' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticCrashLoggingUI' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksModel' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracks' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksEvents' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksTestsObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksTests' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticExperiments' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ Testing failed:
  | [16:45:44]: ▸ 	Could not compute signing task inputs - operation was cancelled.
  | [16:45:44]: ▸ 	Testing cancelled because the build failed.
  | [16:45:44]: ▸ ** TEST FAILED **

A quick Google search for this brings up a few results (e.g. this, this, and this) but all related to running tests via Fastlane on a Swift Package.

The inconsistency

It works on my machine... But not only that, the same commit can pass on one agent and fail on another. For example 10f02d

image

and ef5cc5a

image

I wonder if this has something to do with the agents? Although I don't recall there being any partial update of our fleet. Update: This turned out not to be the case, see details below.

Details

To verify whether we're facing an agent-based issue, I thought we could list failed builds and their agents. If the certain agents consistently pass or fail, that would be an indicator to further investigate the theory.

Agent Failed Build(s) Passed Build(s)
001 https://buildkite.com/automattic/tracks-ios/builds/179
002 https://buildkite.com/automattic/tracks-ios/builds/179 https://buildkite.com/automattic/tracks-ios/builds/177 (notice it was before all other failures, though, 2022/08/26 vs 08/29+ 🤔 )
004 https://buildkite.com/automattic/tracks-ios/builds/167 https://buildkite.com/automattic/tracks-ios/builds/198
007 https://buildkite.com/automattic/tracks-ios/builds/180
013 https://buildkite.com/automattic/tracks-ios/builds/167
014 https://buildkite.com/automattic/tracks-ios/builds/167, https://buildkite.com/automattic/tracks-ios/builds/191

Notice that the table is quite empty right now. This repository doesn't see much action.

Update: As you can see, the MV-MKE-X64-004 agent has a failed and a successful build, so clearly the agent-based theory was wrong.

Dominant language
Swift
Stars
47
Forks
15
Avg merge
1d 34m
Merged PRs (30d)
1

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Automattic/Automattic-Tracks-iOS

All issues in Automattic/Automattic-Tracks-iOS

Similar issues

More Swift issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.