apple/turicreate

Inefficient implementation on checking if path is a s3 directory

Open

#3,049 创建于 2020年3月14日

在 GitHub 查看
 (0 评论) (0 反应) (0 负责人)C++ (11,135 star) (1,161 fork)batch import
S3good first issue

描述

our current implementation of s3 is_directory is to exhaust all keys under a certain URL (a.k.a, prefix, "folder" or "group") and compare the URL with each item returned, e.g., Contents[Key] and CommonPrefixes. This is a bad practice if the URL contains many keys and the comparison is linear O(n).

Besides that, is_directory is widely used in our codebase, not to mention the time complexity, the network delay will be observable.

A solution should be,

  • head on URL assuming it's an object. If 404, then
  • list-objects --with-delimiter and --max-items=1, and check the CommonPrefixes section. You can provide an extra parameter to tell list_objects_impl to do so by setting the max keys to return.

贡献者指南

技术栈
cppaws
领域
performanceinfrastructure
议题类型
performance
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
1-3 hours
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
clear
前置要求
Basic knowledge of AWS S3 operationsFamiliarity with the turicreate codebase structure
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
70
研究方向
Investigate the implementation of `is directory` in the S3 file system code. The current implementation exhaustively lists all keys. Replace it with a two step approach: first HEAD on the URL, then if 404, list objects with delimiter and max items=1, checking CommonPrefixes. The relevant function is likely in the S3 storage adapter file. Ensure the change is efficient and handles edge cases.