drivendataorg/cloudpathlib

Handle `mkdir` for cloud providers that support creating directories

Ouverte

#51 ouverte le 29 août 2020

 (9 commentaires) (1 réaction) (0 personne assignée)Python (87 forks)auto 404
S3design decisionhelp wanted

Métriques du dépôt

Stars
 (624 étoiles)
Métriques de merge PR
 (Métriques PR en attente)

Description

S3 has an interesting situation with folders.

Like other object stores like Azure, it has a flat structure, and when you upload a file to a/b/c.txt for example, it creates an object literally named a/b/c.txt. The directories a and b aren't real and don't exist. The web console has special behavior to fake those as folders in the UI. When you delete c.txt, a and b will automatically be gone.

However, S3 does have another mechanism that lets you have folders. There is a "Create Folders" button in the web console, which lets you make a folders that exist even while empty. These turn out to actually be dummy files with a trailing slash. So if you create a/, there is actually an object in your bucket a/ which is not actually a folder, but the S3 console will treat it like a folder for the UI. You can equivalently upload a file to a/ and it will do the same thing.

We need to think through the implications of this and what cloudpathlib should support.

  • pathlib PurePosixPath strips trailing / on instantiation, so this is something that doesn't map to a representation cleanly through PurePosixPath
  • As a result, it's not currently possible to create an S3Path object that points to an S3 folder. EDIT: This was incorrect. See discussion in #190. This is possible because the string representation of the input URI is the main basis for a CloudPath object, not a PurePosixPath.
  • Additionally, the S3Path.mkdir method is not implemented.

Guide contributeur