2x Improvement requests for the storage migration endpoint

Open

#7,808 opened on Jun 17, 2026 0 comments 0 reactions 0 assignees View on GitHub

@YasenT is already working on this.

Since Jun 17, 2026.

#7807 by @YasenT — open

Assessment

Difficulty: 5/5
Estimated time: Over a week
Newbie friendliness: 25/100

Issue type: Feature
Clarity: Mostly clear
Activity status: Stale
Tech stack: python
Domain: api, backend

Research direction

Start by reviewing the storage migrate endpoint and the linked draft PR #7807, which contains the proposed approach. Compare the current artifact-by-artifact migration and domain-wide locking behavior with the two requested improvements. Done means large-domain migrations support useful concurrency without blocking the entire domain for the full migration.

Written by the indexing model from the issue text.

Description

Feature

I have been using the storage migrate endpoint to switch from OCI storage to S3.
While it worked okay for small domains, when I hit a domain with more artifacts (160K), some flaws popped up.

The migration running artifact by artifact is slow for big domain. Some concurrency might be helpful.
The migration is currently "locking" the whole domain for the duration of it. Which for bigger domains becomes service affecting, not being able to perform anything for 24 hours for example.

I created this draft PR. Feel free to close it if you find the approach bad, it's just something that Claude suggested. https://github.com/pulp/pulpcore/pull/7807

Dominant language: Python
Stars: 598
Forks: 168
Avg merge: 1d 11h
Merged PRs (30d): 95

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

[BUG] GkeTpuCluster queries metadata.google.internal for MEGASCALE_SLICE_ID even when TPU_SKIP_MDS_QUERY is set Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

jax-ml/jax#40854 · Opened Sep 22, 2026
[pcied] Honour PCIE_DETACH_INFO on platforms that are not smartswitches Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

sonic-net/sonic-platform-daemons#900 · Opened Sep 21, 2026
MCP client has no httpx timeout, so the event stream fails after 5 seconds Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

Azure/azure-functions-agents-runtime#228 · Opened Sep 21, 2026
feat(mcp): friendlier error when `run_cloud_sync` is called on a disabled connection Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

airbytehq/PyAirbyte#1205 · Opened Sep 21, 2026
pn7160: reader goes permanently deaf — post-read IRQ-low check fails reads that succeeded Open

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

esphome/esphome#19452 · Opened Sep 21, 2026

More Python issues