set_key and unset_key don't fsync before replacing the .env file
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
Research direction
Start in src/dotenv/main.py at rewrite(), then trace its callers from set_key, unset_key, and the dotenv set/unset commands. Find the existing tests covering rewrite or key updates and add coverage that records fsync and os.replace ordering. Done means the temporary file is durable before replacement, with any directory-durability behavior covered consistently across supported platforms.
Written by the indexing model from the issue text.
Description
set_key and unset_key rewrite the .env file through rewrite() (src/dotenv/main.py): they write a temporary file next to the target, then os.replace it over the target. That makes the swap atomic for other processes, but nothing forces the new contents to disk before the rename. The temporary file is closed (which flushes Python's buffer to the OS) and then renamed, with no fsync.
What can go wrong
If the machine loses power or the kernel crashes shortly after set_key returns, the rename can reach the disk before the file's data does. After a reboot, .env can then be empty (zero length) or contain garbage, instead of holding either the old or the new contents. This applies to set_key, unset_key, and the dotenv set / dotenv unset commands.
How likely this is depends on the filesystem. ext4 with its default auto_da_alloc option flushes data before a rename that replaces an existing file, which covers this exact pattern. Other filesystems (XFS, and ext4 mounted with noauto_da_alloc) make no such promise, and a .env that has never existed before isn't covered by that heuristic at all. I haven't reproduced data loss; this is based on the documented semantics of rename(2) and fsync(2), and is the reason other write-then-rename implementations call fsync.
Since .env files often hold credentials, losing one silently is a bad outcome.
Proposed change
In rewrite(), before os.replace:
dest.flush()
os.fsync(dest.fileno())
This has to happen while the temporary file is still open, i.e. inside the with temp_file as dest: block, after the caller's writes have finished.
Optionally, after os.replace on POSIX, fsync the containing directory so the rename itself is durable:
if os.name == "posix":
dir_fd = os.open(os.path.dirname(os.path.abspath(path)), os.O_RDONLY)
try:
os.fsync(dir_fd)
finally:
os.close(dir_fd)
Directory fsync isn't available on Windows, and some filesystems reject it with EINVAL. Since the file contents are already safe at that point, an OSError from it could reasonably be ignored.
Cost
.env files are small and set_key is not called in hot loops, so one or two fsync calls per write should not be noticeable. A test can check that os.fsync is called on the temporary file before os.replace, for example by patching both in dotenv.main and recording the call order.
Out of scope
Concurrent writers (two set_key calls at once can lose one update) and file-replacement side effects such as single-file Docker bind mounts are separate problems and not addressed here.
- Dominant language
- Python
- Stars
- 8.9k
- Forks
- 612
- Avg merge
- 4d 23h
- Merged PRs (30d)
- 4
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from theskumar/python-dotenv
-
An empty value ignores ${name:-default}Possibly taken @SashaMIT claimed this 11 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
theskumar/python-dotenv#715 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
theskumar/python-dotenv#693 · 1 comment ·
Maintainers usually reply within 1 day
-
python-dotenv installation without cli masks python-dotenv cliPossibly taken @gyanu2507 claimed this 50 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 55/100
theskumar/python-dotenv#683 · 4 comments ·
Maintainers usually reply within 1 day
-
Feature Request: Automatically copy `default.env` to `.env`Possibly taken @dradetsky claimed this 159 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 55/100
theskumar/python-dotenv#644 · 2 comments ·
Maintainers usually reply within 1 day
-
Add `strict` parameter to `load_dotenv()` and `dotenv_values()` for fail-fast behaviorPossibly taken @AhsanSheraz claimed this 211 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 68/100
theskumar/python-dotenv#631 · 3 comments · 1 reaction ·
Maintainers usually reply within 1 day
All issues in theskumar/python-dotenv
Similar issues
-
bug Triage-Needed
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
FuRongJun-1999/dsh-memory#170 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
521xueweihan/HelloGitHub#3940 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100