DBFS append mode silently overwrites existing files
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 77/100
Research direction
Start by reading the DBFS implementation around _initiate_upload and its tests for binary and text append modes. Check how the create request sets overwrite and how unsupported modes are handled. Done means append mode is rejected before opening an upload, with tests confirming existing file contents are not silently overwritten.
Written by the indexing model from the issue text.
Description
DBFS accepts append mode, but _initiate_upload creates the destination with overwrite=True for every mode except xb. Existing contents are discarded, so appending behaves like overwriting.
with fs.open("/existing", "ab") as file:
file.write(b"appended")
I reproduced the request sequence on current master with a mocked DBFS API using its documented create/overwrite behavior. Starting with b"original" leaves only b"appended". The requests are create with overwrite=True, add-block, then close. Text append mode has the same issue.
The DBFS create endpoint supports creating or overwriting a file, and add-block appends to that newly created stream. It does not reopen an existing file for append. Rejecting append mode before opening an upload would prevent silent data loss.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 570
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 102
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from fsspec/filesystem_spec
-
DirCache.items() and values() raise RuntimeError with bounded LRU cachingPossibly taken @Junaid-006 claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
fsspec/filesystem_spec#2419 · 1 comment ·
Maintainers usually reply within 1 day
-
Directory listing cache expiry changes when the system clock jumpsPossibly taken @Dodothereal claimed this 6 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
fsspec/filesystem_spec#2330 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
fsspec/filesystem_spec#2215 · 5 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
fsspec/filesystem_spec#1731 · 3 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
fsspec/filesystem_spec#171 · 4 comments ·
Maintainers usually reply within 1 day
All issues in fsspec/filesystem_spec
Similar issues
-
needs-area-label
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 2 days
-
enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Official-Space-AI/OpenPhysGraspGen#114 ·
Maintainers usually reply within 2 days
-
changelog investigate
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
ramnes/notion-sdk-py#441 ·
-
qa_bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NVIDIA-NeMo/Gym#5644 ·
Maintainers usually reply within 2 days
-
BUG: catalog database credentials printed to logs at startupPossibly taken @cjboyle claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day