drivendataorg/cloudpathlib

Handle `mkdir` for cloud providers that support creating directories

開放

#51 建立於 2020年8月29日

 (9 則留言) (1 個反應) (0 位負責人)Python (87 個分叉)auto 404
S3design decisionhelp wanted

倉庫指標

星標
 (624 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

S3 has an interesting situation with folders.

Like other object stores like Azure, it has a flat structure, and when you upload a file to a/b/c.txt for example, it creates an object literally named a/b/c.txt. The directories a and b aren't real and don't exist. The web console has special behavior to fake those as folders in the UI. When you delete c.txt, a and b will automatically be gone.

However, S3 does have another mechanism that lets you have folders. There is a "Create Folders" button in the web console, which lets you make a folders that exist even while empty. These turn out to actually be dummy files with a trailing slash. So if you create a/, there is actually an object in your bucket a/ which is not actually a folder, but the S3 console will treat it like a folder for the UI. You can equivalently upload a file to a/ and it will do the same thing.

We need to think through the implications of this and what cloudpathlib should support.

  • pathlib PurePosixPath strips trailing / on instantiation, so this is something that doesn't map to a representation cleanly through PurePosixPath
  • As a result, it's not currently possible to create an S3Path object that points to an S3 folder. EDIT: This was incorrect. See discussion in #190. This is possible because the string representation of the input URI is the main basis for a CloudPath object, not a PurePosixPath.
  • Additionally, the S3Path.mkdir method is not implemented.

貢獻者指南