ObjectStore’s maxParallel doesn’t actually increase the number of parallel uploads
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 62/100
Research direction
Start at internalRun in internal/cnpgi/common/wal.go and inspect how it calls GatherReadyWALFiles, using the ten-WAL, maxParallel=3 scenario from the issue as a guide. Done means successive archive_command runs upload distinct batches in parallel-sized groups, avoid re-uploading archived WALs, and exit successfully when no files remain.
Written by the indexing model from the issue text.
Description
Say you have 10 WAL files ready for archiving and maxParallel is set to 3.
- PG runs
archive_commandfor WAL file 1:- barman-plugin looks at ready files and run the archiver for WAL 1, 2 and 3.
- Those 3 files are uploaded to the archive.
- Command exits successfully and PG marks WAL 1 as done.
- Then PG runs
archive_commandfor WAL file 2:- barman-plugin looks are ready file and runs the archiver for WAL 2, 3 and 4
- WAL file 2 and 3 are already archived.
- WAL file 4 is uploaded to the archive.
- Command exits successfully and PG marks WAL 2 as done.
- And so on, PG archives WAL file 3 and barman-plugin uploads WAL file 5.
This results in always uploading one single WAL file at a time which is pretty slow and is contrary to what the doc says: “Number of WAL files to be […] archived in parallel”.
What I would expect to happen is:
- PG runs
archive_commandfor WAL file 1:- WAL file 1, 2 and 3 are uploaded
- WAL file 1 is marked as done
- PG runs
archive_commandfor WAL file 2:- WAL file 4, 5, 6 are uploaded
- WAL file 2 is marked as done
- And so on until there are no more files to upload and the
archive_commandsimply exits successfully without doing anything.
I think this happens because internalRun does not pass WALs that have already been archived to GatherReadyWALFiles: https://github.com/cloudnative-pg/plugin-barman-cloud/blob/376e178ab5ea907aae50e6eaeb19215ec4326c53/internal/cnpgi/common/wal.go#L206.
- Dominant language
- Go
- Stars
- 192
- Forks
- 75
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 17
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cloudnative-pg/plugin-barman-cloud
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
cloudnative-pg/plugin-barman-cloud#1104 · 4 reactions ·
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
cloudnative-pg/plugin-barman-cloud#1113 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 15/100
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
All issues in cloudnative-pg/plugin-barman-cloud
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 60/100
github/gh-aw-mcpg#13748 ·
-
agentic-workflows
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
googleapis/librarian#7670 · 2 comments ·