4 stage upload process

Open
#274 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
cli

Research direction

The issue describes ddsclient's existing two-stage upload process but names no files, tests, or entry points. Trace the upload command and its scheduling logic, then determine how files are classified as small or large and new or existing. Done means the process handles four ordered stages: small new, large new, small existing, and large existing.

Written by the indexing model from the issue text.

Description

Yes. The way ddsclient uploads files is a two stage process. The first stage we upload small files - several at a time. The second stage we upload large files one at a time (with chunks be uploaded in parallel). Currently we sort the new files to the front within each stage.
Should I break it up into 4 stages: upload small new, upload large new, upload small existing, upload large existing?

Eventually yes, but let's create an issue for that item.

Originally posted by @dleehr in https://github.com/Duke-GCB/DukeDSClient/pull/271#issuecomment-540049128

Dominant language
Python
Stars
5
Forks
6
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Duke-GCB/DukeDSClient

All issues in Duke-GCB/DukeDSClient

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.