open-guides/og-aws

S3: Discuss ways to list and tally objects efficiently

Open

#58 opened on Sep 2, 2016

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Shell (36,412 stars) (3,905 forks)batch import
help wantedunder discussion

Description

Topics:

  • Listing and pagination
  • Need for multi-threaded S3 crawl over keys for speed
    • Prefix-based listings, with separators
    • Hash-type prefixes with known alphabet, uniform distribution
  • Possibly: Reassigning work; using markers to optimize if alphabet is not known
  • Tallying usage by mapreduce over keys that propagate usage up by folder

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

Contributor guide

Tech stack
aws
Domain
cloudperformance
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
AWS S3 basicsunderstanding of paginationdistributed systems concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Review the linked AWS documentation (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html) and explore existing comments for specific optimization strategies. Investigate multi threaded crawling with hash based prefixes and consider how to implement a tallying mechanism via map reduce on keys. Look for any existing implementations or discussions in the repository.