Handle path-style object sotrage HTTPS URLs correctly in ObjectStoreRegistry.resolve
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
Research direction
Start with ObjectStoreRegistry.resolve() and its url-handling branch, then compare the mounted AzureStore prefix with the path-style HTTPS URL in the reproduction. Add regression coverage for the Azure and path-style S3 cases, and verify that non-object-storage HTTPS and virtual-hosted S3 resolution remain unchanged.
Written by the indexing model from the issue text.
Description
ObjectStoreRegistry.resolve() returns the wrong trailing path for path-style object storage HTTPS URLs when the registered store is already mounted to a container or bucket.
In these setups, the resolved path still includes the container or bucket name, but the underlying store already treats that container or bucket as the storage root. Passing the resolved path into store.head() or store.get_range() therefore targets a non-existent object.
Azure Blob Storage HTTPS URLs are a clear real-world example of this bug. The same class of bug can also affect path-style S3 HTTPS URLs.
Current behavior
Given an Azure Blob asset URL like:
https://ukmoeuwest.blob.core.windows.net/deterministic/uk/near-surface/...nc
and a store created from the same container:
store = AzureStore(
credential_provider=PlanetaryComputerCredentialProvider(
"https://ukmoeuwest.blob.core.windows.net/deterministic/"
)
)
registry = ObjectStoreRegistry(
{"https://ukmoeuwest.blob.core.windows.net/deterministic/": store}
)
registry.resolve(asset_url) returns:
"deterministic/uk/near-surface/...nc"
But AzureStore expects a path within the deterministic container:
"uk/near-surface/...nc"
The extra deterministic/ causes store.head() to 404.
Expected behavior
For path-style object storage HTTPS URLs, ObjectStoreRegistry.resolve() should return a path that is compatible with the mounted store, i.e. a path relative to the configured container or bucket.
For the Azure example above, the resolved path should be:
"uk/near-surface/...nc"
Steps to reproduce
Minimal repro with current behavior:
from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider
from obstore.store import AzureStore
from obspec_utils.registry import ObjectStoreRegistry
prefix = "https://ukmoeuwest.blob.core.windows.net/deterministic/"
asset_url = (
"https://ukmoeuwest.blob.core.windows.net/"
"deterministic/uk/near-surface/20260501T0000Z/"
"20260501T0000Z-PT0000H00M-temperature_at_surface.nc"
)
store = AzureStore(
credential_provider=PlanetaryComputerCredentialProvider(prefix)
)
registry = ObjectStoreRegistry({prefix: store})
_, path = registry.resolve(asset_url)
print(path)
# current output:
# deterministic/uk/near-surface/20260501T0000Z/
# 20260501T0000Z-PT0000H00M-temperature_at_surface.nc
print(store.head("uk/near-surface/20260501T0000Z/20260501T0000Z-PT0000H00M-temperature_at_surface.nc")["size"])
# works
print(store.head(path)["size"])
# FileNotFoundError / 404
A smaller non-network repro for the path resolution itself:
from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider
from obstore.store import AzureStore
from obspec_utils.registry import ObjectStoreRegistry
prefix = "https://ukmoeuwest.blob.core.windows.net/deterministic/"
asset_url = (
"https://ukmoeuwest.blob.core.windows.net/"
"deterministic/uk/near-surface/20260501T0000Z/"
"20260501T0000Z-PT0000H00M-temperature_at_surface.nc"
)
store = AzureStore(
credential_provider=PlanetaryComputerCredentialProvider(prefix)
)
registry = ObjectStoreRegistry({prefix: store})
_, path = registry.resolve(asset_url)
assert path == "uk/near-surface/20260501T0000Z/20260501T0000Z-PT0000H00M-temperature_at_surface.nc"
# currently fails because path includes the container name
Impact
This breaks downstream code that relies on ObjectStoreRegistry.resolve() to feed resolved paths into mounted object-store methods.
One concrete case is VirtualiZarr, where the resolved path is passed into a buffered reader backed by AzureStore.head() and AzureStore.get_range(). The dataset open fails even though the original HTTPS asset href is valid and the store is correctly authenticated.
This does not appear to affect every HTTPS object URL shape. For example, virtual-hosted S3 URLs like https://my-bucket.s3.us-west-2.amazonaws.com/... appear to resolve correctly because the bucket name lives in the hostname rather than in the path. The problem shows up when the storage identity is encoded in the path, as with Azure Blob HTTPS URLs and path-style S3 URLs like https://s3.us-west-2.amazonaws.com/my-bucket/....
Notes / possible cause
The issue appears to come from this branch in ObjectStoreRegistry.resolve():
elif hasattr(store, "url"):
prefix = urlparse(store.url).path.lstrip("/")
path_after_prefix = path.lstrip("/").removeprefix(prefix).lstrip("/")
else:
path_after_prefix = path.lstrip("/")
AzureStore does not expose .url, and the registry falls back to returning the full object path from the HTTPS URL.
For Azure Blob HTTPS URLs, that full path includes the container name. But AzureStore already knows the container and expects paths relative to it.
The same mismatch can arise for other path-style HTTPS object URLs where the bucket or container lives in the URL path but the mounted store already knows that storage root.
This suggests the bug is not Azure-specific. It is a mismatch between path-style HTTPS URL resolution and stores that are already mounted to a bucket or container.
Acceptance criteria
-
ObjectStoreRegistry.resolve()returns container-relative or bucket-relative paths for path-style object storage HTTPS URLs when resolving against mounted stores - A regression test covers an
AzureStoreregistered under ahttps://<account>.blob.core.windows.net/<container>/prefix - A regression test covers the equivalent path-style S3 case, such as
https://s3.<region>.amazonaws.com/<bucket>/... - Existing non-object-storage HTTP/HTTPS resolution behavior remains unchanged
- Existing virtual-hosted object-storage HTTPS behavior remains unchanged
Open questions
- Should this be handled by teaching
resolve()about mounted object stores with path-style HTTPS URLs, or by exposing enough metadata on store objects to make the existing prefix-stripping logic work generically? - Should
https://<account>.blob.core.windows.net/<container>/...andabfs://<container>/...be normalized to equivalent resolved paths in the registry? - Should the same normalization apply to path-style S3 HTTPS URLs so they behave like
s3://bucket/...and virtual-hosted S3 HTTPS URLs?
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from developmentseed/obspec-utils
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
developmentseed/obspec-utils#93 · 1 comment ·
-
resolve() truncates object keys containing '#', '?', or ';' (breaks special-char keys and presigned URLs)Possibly taken @TomNicholas claimed this 84 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Switch docs hosting to developmentseed domainMay be free again @maxrjones claimed this 165 days ago, and no pull request is open. Open
developmentseed/obspec-utils#67 · 1 comment · 1 assignee ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 52/100
developmentseed/obspec-utils#63 · 2 reactions ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
developmentseed/obspec-utils#62 · 1 comment ·
All issues in developmentseed/obspec-utils
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 60/100
521xueweihan/HelloGitHub#3924 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
wilbowes/EchoMuse#869 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
namespace operations
Difficulty 1/5 Under an hour Newbie friendliness 72/100
EclipseFdn/open-vsx.org#14043 ·
Maintainers usually reply within 1 day
-
test: TestServeUntilStale races the server's close against the client's sendall (BrokenPipeError under load)Possibly taken @evoludigit claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 89/100
Maintainers usually reply within 1 day