settings.py flaps between reconciles: needsMigrationSetting depends on map iteration order
Maintainers usually reply within 4 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
- Domain
- infrastructure
Research direction
Read controllers/repo_manager/secret.go, starting with needsMigrationSetting() and how it uses MigrationSettingsList(). Check for related tests before making changes. Done means the generated settings.py is stable across reconciles and every setting set to true is written.
Written by the indexing model from the issue text.
Description
Version
Operator 1.3.0. The same code is on main.
Describe the bug
needsMigrationSetting() in controllers/repo_manager/secret.go builds part of settings.py by ranging over MigrationSettingsList(), which is a Go map, and it returns at the first setting that is false.
Go randomizes map iteration order. With redirect_to_object_storage: true and hide_guarded_distributions: false, REDIRECT_TO_OBJECT_STORAGE = True is written on some reconciles and missing on others. With both settings true, the two lines swap order.
So the <name>-server Secret changes between reconciles. The operator logs The Data from Secret pulp-server has been modified! Reconciling ... and Reprovisioning pulpcore pods to get the new settings ..., and rolls the api, content and worker Deployments. The rollout triggers more reconciles, each with roughly a 50% chance of flipping the Secret again, so the loop sustains itself. We saw it run for minutes after a single CR change, and once it ran for days, creating tens of thousands of ReplicaSets.
There is a second effect. Depending on the order, a true setting can be dropped from settings.py entirely, so REDIRECT_TO_OBJECT_STORAGE is sometimes not applied.
To Reproduce
- Set
redirect_to_object_storage: trueand leavehide_guarded_distributionsunset. - Change anything on the CR to trigger a reconcile.
- Watch the operator logs and
kubectl get rs.
Expected behavior
settings.py is identical on every reconcile, and each true setting is written.
Additional context
Fix: iterate over sorted keys, and continue instead of return on a false setting. PR to follow.
- Dominant language
- Go
- Stars
- 88
- Forks
- 68
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 1
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pulp/pulp-operator
-
Documentation for metadata signing (APT/RPM) removed but still relevantMay be free again A pull request for this issue was closed without being merged. OpenIssue Triage-Needed
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pulp/pulp-operator#1644 ·
Maintainers usually reply within 4 days
-
Issue Triage-Needed
Difficulty 3/5 1-2 days Newbie friendliness 74/100
pulp/pulp-operator#1685 ·
Maintainers usually reply within 4 days
-
Feature Triage-Needed
Difficulty 5/5 Over a week Newbie friendliness 35/100
pulp/pulp-operator#1682 ·
Maintainers usually reply within 4 days
-
Issue Triage-Needed
Difficulty 3/5 1-2 days Newbie friendliness 45/100
pulp/pulp-operator#1647 ·
Maintainers usually reply within 4 days
-
Job definition for reset-admin-password doesn't propagate image pull policyMay be free again A pull request for this issue was closed without being merged. OpenIssue
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
pulp/pulp-operator#1630 ·
Maintainers usually reply within 4 days
All issues in pulp/pulp-operator
Similar issues
-
Helm IPv4 host checks accept addresses Go rejectsPossibly taken @ericcaiwx-star claimed this today. Openclawsweeper:bulk-filed clawsweeper:linked-pr-open clawsweeper:no-new-fix-pr clawsweeper:source-repro impact:other issue-rating: 🦞 diamond lobster P2
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
openclaw/openclaw-enterprise#1588 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
cvss-severity:high devguard l3montree-cybersecurity/devguard/devguard-web pkg:oci/devguard-web?rep...ch=amd64&tag=main-amd64 pkg:oci/devguard-web?rep...ch=arm64&tag=main-arm64 pkg:oci/web?repository_u...ch=amd64&tag=main-amd64 pkg:oci/web?repository_u...ch=arm64&tag=main-arm64 risk:low state:open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
l3montree-dev/devguard#3168 · 1 comment ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
status:approved type:bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Gentleman-Programming/gentle-ai#5326 ·
Maintainers usually reply within 1 day
-
needs-acceptance wg/router-models-inference-runtime
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
vllm-project/semantic-router#4663 ·
Maintainers usually reply within 1 day