open-guides/og-aws

S3: Discuss ways to list and tally objects efficiently

Open

#58 建立於 2016年9月2日

在 GitHub 查看
 (9 留言) (0 反應) (0 負責人)Shell (36,412 star) (3,905 fork)batch import
help wantedunder discussion

描述

Topics:

  • Listing and pagination
  • Need for multi-threaded S3 crawl over keys for speed
    • Prefix-based listings, with separators
    • Hash-type prefixes with known alphabet, uniform distribution
  • Possibly: Reassigning work; using markers to optimize if alphabet is not known
  • Tallying usage by mapreduce over keys that propagate usage up by folder

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

貢獻者指南

技術棧
aws
領域
cloudperformance
議題類型
research
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
4
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
over 1 week
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
needs investigation
前置要求
AWS S3 basicsunderstanding of paginationdistributed systems concepts
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
20
研究方向
Review the linked AWS documentation (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html) and explore existing comments for specific optimization strategies. Investigate multi threaded crawling with hash based prefixes and consider how to implement a tallying mechanism via map reduce on keys. Look for any existing implementations or discussions in the repository.
S3: Discuss ways to list and tally objects efficiently · open-guides/og-aws#58 | Good First Issue