open-guides/og-aws

S3: Discuss ways to list and tally objects efficiently

Open

#58 opened on Sep 2, 2016

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Shell (3,905 forks)batch import
help wantedunder discussion

Repository metrics

Stars
 (36,412 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Topics:

  • Listing and pagination
  • Need for multi-threaded S3 crawl over keys for speed
    • Prefix-based listings, with separators
    • Hash-type prefixes with known alphabet, uniform distribution
  • Possibly: Reassigning work; using markers to optimize if alphabet is not known
  • Tallying usage by mapreduce over keys that propagate usage up by folder

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

Contributor guide

Research direction
Study the S3 REST API for listing objects with pagination, explore multi threaded crawling strategies using prefix based parallelism, and design a mapreduce like approach to tally usage by folder. Reassigning work using markers may be needed for non uniform key distributions.
Tech stack
None
Domain
cloud
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
AWSS3
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30