akka/akka-http

Merge stream elements into one http chunk

Aperta

#1792 aperta il 17 gen 2018

 (4 commenti) (0 reazioni) (0 assegnatari)Scala (598 fork)batch import
1 - triageddiscusshelp wantedt:http:marshallingt:routingt:stream

Metriche repository

Star
 (1311 stelle)
Metriche merge PR
 (Merge medio 1g 10h) (2 PR mergiate in 30 g)

Descrizione

Streams are great but I got a stream of thousand elements. Unfortunately every element is emitted in one http chunk (and the elements are small). Would be nice if the framework supports "merging" of multiple elements into one chunk and return a valid json array of elements.

My workaround

                      def toJsonBytes(source: Seq[T]): ByteString =
                        ByteString(source.map(t => t.asJson.noSpaces).mkString(","))
                      // emit 100 elements in one http chunk
                      val chunk = t.groupedWithin(100, 100.millis)
                      val fst   = Flow[Seq[T]].map(toJsonBytes)
                      val rst   = Flow[Seq[T]].map(ByteString(",") ++ toJsonBytes(_))
                      val together = chunk.prefixAndTail(1).flatMapConcat {
                        case (head, tail) =>
                          Source(head).via(fst).concat(tail.via(rst))
                      }
                      complete(
                        HttpEntity(
                          ContentTypes.`application/json`,
                          Source.single(ByteString("[")) ++ together ++ Source.single(ByteString("]"))
                        )
                      )

Guida contributor