Cubestore s3 fails when uploading with`File can't be listed after upload.` when subPath starts with / · cube-js/cube#6136

(1 comment) (1 reaction) (0 assignees)Rust (19,563 stars) (1,965 forks)batch import

cube storehelp wanted

Description

Describe the bug Errors like:

Error during Metastore upload: CubeError { message: "File metastore-1675803621852-logs/2.flex can't be listed after upload. Either there's Cube Store cluster misconfiguration, or storage can't provide the required consistency.", backtrace: "", cause: Internal }

for every uploaded file to s3 if subpath starts with /

To Reproduce Steps to reproduce the behavior: Deploy cubestore with AWS s3 storage and use a subpath with leading /

Expected behavior Error shouldnt happen as AWS now provides strong read-after-write consistency

Screenshots Check related slack thread

Minimally reproducible Cube Schema

Version: master

Additional context Related rust-s3 open issue: https://github.com/durch/rust-s3/issues/203

I fixed it by removing the leading / on my subpath, but cubestore should be able to manage this case silently.

Contributor guide

Tech stack: rustaws
Domain: backendcloud
Issue type: bug
Difficulty: 2
Estimated time: under 1 hour
Activity status: fresh
Clarity: clear
Prerequisites: RustAWS S3 basics
Newbie friendliness: 85
Research direction: Investigate the Cubestore S3 storage code, likely in the Rust source files. Search for where the subpath configuration is used during file listing or uploading. The fix involves stripping a leading slash from the subpath if present. Review the related rust s3 issue (https://github.com/durch/rust s3/issues/203) for insights. Ensure the change handles both cases (with and without leading slash) without breaking existing functionality.