twitter/finatra

File Upload as a Stream

Open

#538 opened on Apr 27, 2020

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Scala (2,274 stars) (403 forks)batch import
help wanted

Description

Finatra currently does not support streaming files uploaded via a multipart HTTP request.

Expected behavior

The ability to stream the raw bytes of a file being uploaded using com.twitter.finatra.http.fileupload.FinagleRequestFileUpload

Actual behavior

As per the TODO inside of com.twitter.finatra.http.fileupload.MultipartItem: // TODO don't store file in memory; write to temp file and store file reference.

Currently, the entire file is read into memory inside of parseMultipartItems in FinagleRequestFileUpload

Steps to reproduce the behavior

Self-explanatory.

Attempted resolution(s)

To say I'm a beginner with Scala and Finatra/Finagle would be giving me too much credit... I tried to extend FinagleRequestFileUpload with my own class and create my own version of MultipartItem to get the behaviors that I want, but there must be some magic going on that I'm not aware of.

When I try:

  post("/upload") { request: MyRequestFileUpload =>
   // where `fromRequest()` is my version of `FinagleRequestFileUpload.parseMultipartItems()`
    request.fromRequest(request)
  }

I get the following compile time exception:

Error:(23, 25) type mismatch;
 found   : com.twitter.finatra.http.fileupload.MyRequestFileUpload
 required: com.twitter.finagle.http.Request
    request.fromRequest(request)

If there is any way at all that I can be of assistance, please let me know. I would love to use Finatra, as it's much higher-level than Finagle, but I am required to upload files that could be in excess of several gigabytes and don't really want to do anything too weird.

If you know of a work-around that I could do, please let me know.

Thank you all for the awesome library!

EDIT:

After some more digging, I found out that my implementation should be:

  post("/upload") { request: Request =>
   // where `fromRequest()` is my version of `FinagleRequestFileUpload.parseMultipartItems()`
    val map: Map[String, MyMultipartItem] = new MyRequestFileUpload().fromRequest(request)
  }

Now, the hard part... how to properly turn this into a stream 🙃

EDIT 2: Maybe best to use this Netty 4 example for inspiration

Contributor guide

Tech stack
scala
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Scala basicsFinatra frameworkFinagle internalsHTTP multipart
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The main files to modify are FinagleRequestFileUpload.scala (parseMultipartItems method) and MultipartItem.scala (which currently stores file in memory). The TODO in MultipartItem suggests writing to a temp file. Look at the Netty 4 example linked in the issue for streaming approach. Also review the author's attempt to extend the classes for guidance.