Changes to ObjectStore spec do not get applied until after the cluster is healthy
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
Research direction
The issue names ObjectStore, instanceSidecarConfiguration, the barman-cloud-plugin operator, and a stuck 1/2 Ready replica; start by tracing how that configuration is applied when a pod is unhealthy. Reproduce the AWS checksum failure with the shown environment variables and a blocked replica, then verify that the outstanding configuration is applied without manually deleting the pod.
Written by the indexing model from the issue text.
Description
We had an issue yesterday where we updated QNAP QuObjects, causing some change to the way S3 is handled. The resultant error from barman cloud was:
ERROR: Barman cloud WAL archiver exception: An error occurred (InvalidDigest)
when calling the PutObject operation: The Content-MD5 or checksum value that
you specified is not valid.
The fix is already mentioned in the documentation, where we add the following to the object store:
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
spec:
configuration:
...
instanceSidecarConfiguration:
env:
- name: AWS_REQUEST_CHECKSUM_CALCULATION
value: when_required
- name: AWS_RESPONSE_CHECKSUM_VALIDATION
value: when_required
and I can confirm this works perfectly. However. Because one of the pods is stuck (1/2 Ready, blocked on the error above) the update never gets applied. What I had to do was apply the fix and then manually delete that one stuck pod (3-node cluster, this pod was a replica).
I think this is a bug, just because this configuration change was specifically required to resolve the issue I was having (S3 semantics changed). It's a little more complicated, here, because it seems like I had a primary switchover (perhaps due to similar S3-related errors) which left a bunch of WALs waiting to get uploaded. Would be good to see if there's a way this could be improved such that the barman-cloud-plugin operator is able to detect the S3-related failure and apply outstanding configuration.
- Dominant language
- Go
- Stars
- 192
- Forks
- 75
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 17
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cloudnative-pg/plugin-barman-cloud
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
cloudnative-pg/plugin-barman-cloud#1104 · 4 reactions ·
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
cloudnative-pg/plugin-barman-cloud#1113 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 15/100
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
All issues in cloudnative-pg/plugin-barman-cloud
Similar issues
-
ai-generated
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
SiaFoundation/indexd#1116 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
wakatime/wakatime-cli#1585 ·