drivendataorg/cloudpathlib

Handle `mkdir` for cloud providers that support creating directories

Offen

#51 geöffnet am 29.08.2020

 (9 Kommentare) (1 Reaktion) (0 zugewiesene Personen)Python (87 Forks)auto 404
S3design decisionhelp wanted

Repository-Metriken

Stars
 (624 Sterne)
PR-Merge-Metriken
 (PR-Metriken ausstehend)

Beschreibung

S3 has an interesting situation with folders.

Like other object stores like Azure, it has a flat structure, and when you upload a file to a/b/c.txt for example, it creates an object literally named a/b/c.txt. The directories a and b aren't real and don't exist. The web console has special behavior to fake those as folders in the UI. When you delete c.txt, a and b will automatically be gone.

However, S3 does have another mechanism that lets you have folders. There is a "Create Folders" button in the web console, which lets you make a folders that exist even while empty. These turn out to actually be dummy files with a trailing slash. So if you create a/, there is actually an object in your bucket a/ which is not actually a folder, but the S3 console will treat it like a folder for the UI. You can equivalently upload a file to a/ and it will do the same thing.

We need to think through the implications of this and what cloudpathlib should support.

  • pathlib PurePosixPath strips trailing / on instantiation, so this is something that doesn't map to a representation cleanly through PurePosixPath
  • As a result, it's not currently possible to create an S3Path object that points to an S3 folder. EDIT: This was incorrect. See discussion in #190. This is possible because the string representation of the input URI is the main basis for a CloudPath object, not a PurePosixPath.
  • Additionally, the S3Path.mkdir method is not implemented.

Contributor Guide