playframework - how does BodyParser.Raw in Play! framework work?

Question

I have a question about the BodyParser.Raw in Play! framework. The official document says:

Parses the body as a RawBuffer. This will attempt to store the body in memory, up to Play’s configured memory buffer size, but fallback to writing it out to a File if that’s exceeded.

I am having a hard time understanding the above description. Does this mean the following -

If I put the following configurations to my application.conf file:

play.http.parser.maxMemoryBuffer=128k
play.http.parser.maxDiskBuffer=1G

and say my request body has a size of 15MB, then Play! will read/parse the first 128k from the request body, write it to a file, and then stream/parse the next 128k from the request body, write it to the same file, ..., until the 15MB request body is fully parsed? This means that we are using only 128k memory, and we are able to handle a request body whose size is 15MB??

I know it sounds too good to be true...

Could someone kindly explain this?

UPDATE:

I want to add some updates here if someone has encountered the same problem here. So @Ivan Kurchenko is right (see below his answer): BodyParser.Raw can be used to POST a request body that is larger than the configured play.http.parser.maxMemoryBuffer. Play! will buffer the request using the disk in that case. In other words, you can POST a big request body that is even larger then the available memory in your JVM.

The reason my test failed using Chrome (and also Firefox) was because I had CSRF filter enabled by default. Once I disabled it, everything works fine. For example, the following configurations,

play.http.parser.maxMemoryBuffer=1k
play.http.parser.maxDiskBuffer=4G
play.filters.disabled+=play.filters.csrf.CSRFFilter

says if the request body is larger than 1k, Play! will use your disk to buffer your request (and certainly, I disable the CSRF filter). Now, with this configuration, I can successfully upload a movie which has a size of more than 3G. So indeed, BodyParser.Raw can be used if you need to handle a large request. HTH!

question from:https://stackoverflow.com/questions/65894923/how-does-bodyparser-raw-in-play-framework-work

Answer 1

Not really - if request size exceeds memory threshold, in your case configured as play.http.parser.maxMemoryBuffer=128k whole requests body will be written to a temporary file or written to a memory otherwise. And then for the next parsing body will be also streamed either from file or from memory. Let's look together:

BodyParser.raw - creates RawBuffer inside, like for instance here https://github.com/playframework/playframework/blob/6d0789468909d5d7bdabc6c4207337bd7a7ca9b1/core/play/src/main/scala/play/api/mvc/BodyParsers.scala#L612

And inside RawBuffer two methods we interested in : push method implementation - which accepts incoming bytes and total size exceed buffer size execute backToTemporaryFile - meaning write all request to a temporary file.

  @volatile private var inMemory: ByteString                 = initialData
  @volatile private var backedByTemporaryFile: TemporaryFile = _
  @volatile private var outStream: OutputStream              = _

  private[play] def push(chunk: ByteString): Unit = {
    if (inMemory != null) { //checks whether current in memory buffer exists

      //if next readed chunk of request body exceeds in memory buffer size
      if (chunk.length + inMemory.size > memoryThreshold) {
        backToTemporaryFile()  // create temporary file
        outStream.write(chunk.toArray)// write buffer to temporary file
      } else {
        inMemory = inMemory ++ chunk // append in memory buffer with next chunk
      }
    } else {
      outStream.write(chunk.toArray) // append in memory buffer with next chunk
    }
  }

And then asBytes method implementation - which on its own side read bytes if in memory buffer exists. Or asFile implementation which reads bytes from temporary file.

Also Note: If incoming request body size exceeds configured value in play.http.parser.maxDiskBuffer Play will respond with 413 - REQUEST_ENTITY_TOO_LARGE status.