Snap memory usage for large HTTP response bodies

Reproduce the two cases using the issue's writeLBS, readRequestBody, and quickHttpServe entry points, comparing memory behavior for the large lazy ByteString. Trace the response and chunked-encoding path to identify where buffering occurs; done means large responses are delivered with bounded memory while preserving the observed streaming behavior.

Hi, I'm trying to deliver a large HTTP response using the Haskell Snap framework, but memory usage grows in proportion to the size of the response. Here's a couple of cut down test cases that use a large lazy ByteString:

import Snap.Core (Snap, writeLBS, readRequestBody)
import Snap.Http.Server (quickHttpServe)
import Control.Monad.IO.Class (MonadIO(liftIO))
import qualified Data.ByteString.Lazy.Char8 as LBS (ByteString, length, replicate)

main :: IO ()
main = quickHttpServe $ site test1 where
    test1, test2 :: LBS.ByteString -> Snap ()

    -- Send ss to client
    test1 = writeLBS

    -- Print ss to stdout upon receiving request
    test2 = liftIO . print

    site write = do
        body <- readRequestBody 1000
        -- Making ss dependant on the request stops GHC from keeping a
        -- reference to ss as pointed out by Reid Barton.
        let bodyLength = fromIntegral $ LBS.length body
        write $ ss bodyLength

    ss c = LBS.replicate (1000000000000 * (c + 1)) 'S'

• test1 delivers ss to the client. Memory usage grows in proportion to the ByteString's size.
• test2 prints ss to stdout within the Snap monad stack upon receiving a request. This runs in a small constant amount of memory.

The responses are delivered using chunked encoding (I checked, and it's 1Tb so it would have to be). I thought Snap should also be able to deliver the response in constant memory. Is there any way to achieve this? It's also worth noting that the response starts being delivered immediately.

Memory was measured using "top" on linux and was observed to grow to 15GB resident at which point it was just starting to swap. The memory grew by jumping a factor of 2 each time rather than steadily increasing. The request did complete and deliver ~1Terabyte, so the memory usage is a couple of orders of magnitude less than the size of the ByteString. Once the request had completed, it sat at 15Gb resident (I left it for a few minutes). When I fired another request at it, it still remained steady at 15Gb and completed the request as before. The virtual size stayed within 5% of the resident size.

Firing 2 concurrent requests at it resulted at first in a drop in virtual and resident memory to about 5Gb, followed by an increase to about 17Gb at which point the machine was getting unusable so I killed the process.

GHC version 7.8.3, Snap version 0.14.0.5

Also GHC version 7.10.2, Snap version 0.14.0.6