SFTP put_file passes full destination URLs to Paramiko
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 83/100
Research direction
Start in the SFTPFileSystem.put_file implementation in fsspec/implementations/sftp.py and compare how it normalizes the destination for makedirs with the path passed to Paramiko.put. Check the SFTP tests for existing URL and ssh alias coverage, then add or update cases to verify both forms send the normalized remote path. Done when file transfer and parent creation use the same normalized destination.
Written by the indexing model from the issue text.
Description
Found while reviewing remote upload paths. SFTPFileSystem.put_file normalizes the destination parent through makedirs, but then passes the original full URL to Paramiko.put as the remote filename.
Reproduced on current master, Python 3.12 and Paramiko 5.1.0 with a filesystem-backed transport double. Other public SFTP methods already strip the protocol.
from unittest.mock import patch, Mock
from fsspec.implementations.sftp import SFTPFileSystem
with patch.object(SFTPFileSystem, "_connect"):
fs = SFTPFileSystem(host="host", skip_instance_cache=True)
fs.ftp = Mock()
fs.mkdirs = Mock()
fs.put_file("source.txt", "sftp://host/parent/target.txt")
print(fs.ftp.put.call_args)
The destination sent to Paramiko is sftp://host/parent/target.txt rather than /parent/target.txt. The ssh alias has the same problem. A server then interprets the URL as a literal relative path and the upload fails, despite the correct parent having been created.
Expected the same normalized destination for parent creation and file transfer.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 577
- Avg merge
- 2d 36m
- Merged PRs (30d)
- 51
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from fsspec/filesystem_spec
-
FTP reads inside a transaction use a nonexistent staging pathPossibly taken @Harsh23Kashyap claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
fsspec/filesystem_spec#2463 ·
Maintainers usually reply within 1 day
-
GistFileSystem.cat ignores start and end read boundsPossibly taken @IlyaKhalafi claimed this 9 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
fsspec/filesystem_spec#2438 ·
Maintainers usually reply within 1 day
-
DBFS append mode silently overwrites existing filesPossibly taken @IlyaKhalafi claimed this 11 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
fsspec/filesystem_spec#2429 ·
Maintainers usually reply within 1 day
-
DirCache.items() and values() raise RuntimeError with bounded LRU cachingPossibly taken @draarivpatel-ui claimed this 3 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
fsspec/filesystem_spec#2419 · 1 comment ·
Maintainers usually reply within 1 day
-
Directory listing cache expiry changes when the system clock jumpsPossibly taken @Dodothereal claimed this 16 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
fsspec/filesystem_spec#2330 ·
Maintainers usually reply within 1 day
All issues in fsspec/filesystem_spec
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
openvinotoolkit/openvino_notebooks#3858 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 88/100
hummingbot/hummingbot-api#256 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
area:other help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
WingedGuardian/GENesis-AGI#4734 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
peach-mitao/peach#693 ·
Maintainers usually reply within 1 day