java - Impact of setting -XX:+DisableExplicitGC when NIO direct buffers are used

Question

We are building a web application with aggressive performance SLAs which are periodically being violated due to the JVM going out to lunch due to System.gc() calls. We've done some debugging, and determined that in all cases it's the internal app server code which is invoking System.gc(). This occurs a few times while the app server is booting or the application is deployed, which we aren't concerned about. However, System.gc() is also periodically be triggered when the app is up and running via internal app server calls the NIO classes. Here's a stack trace we were able to capture of this event:

3XMTHREADINFO      "WebContainer : 25" J9VMThread:0x0000000006FC5D00, j9thread_t:0x00007F60E41753E0, java/lang/Thread:0x000000060B735590, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0xFE, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x1039, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range from:0x00007F6067621000, to:0x00007F6067662000, size:0x41000)
3XMCPUTIME               CPU usage total: 80.222215853 secs
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=1594568 (0x1854C8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/System.gc(System.java:329)
4XESTACKTRACE                at java/nio/Bits.syncReserveMemory(Bits.java:721)
5XESTACKTRACE                   (entered lock: java/nio/Bits@0x000000060000B690, entry count: 1)
4XESTACKTRACE                at java/nio/Bits.reserveMemory(Bits.java:766(Compiled Code))
4XESTACKTRACE                at java/nio/DirectByteBuffer.<init>(DirectByteBuffer.java:123(Compiled Code))
4XESTACKTRACE                at java/nio/ByteBuffer.allocateDirect(ByteBuffer.java:306(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateBufferDirect(WsByteBufferPoolManagerImpl.java:706(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateCommon(WsByteBufferPoolManagerImpl.java:612(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateDirect(WsByteBufferPoolManagerImpl.java:527(Compiled Code))
4XESTACKTRACE                at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:507(Compiled Code))
4XESTACKTRACE                at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1864(Compiled Code))
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00007F61083DD122 [libj9prt26.so+0x13122])
4XENATIVESTACK               (0x00007F61083EA79F [libj9prt26.so+0x2079f])
....

Is anyone aware of what the impact would be if we shut down the calls to System.gc() by enabling -XX:+DisableExplicitGC (or actually in our case by setting -Xdisableexplicitgc since we're running Websphere on the IBM JRE, which does the same thing)? We certainly don't want to create a memory leak. I haven't been able to find a direct reference as to why the System.gc() calls in NIO are actually necessary, and there isn't a code comment specifically addressing it where it occurs in JDK code, either: http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/4a1e42601d61/src/share/classes/java/nio/Bits.java

If it's a bad idea to completely disable System.gc() due to the use of NIO, is there at least something we can do to reduce the frequency at which it's called? It appears that we can set -XX:MaxDirectMemorySize, but this appears as if it would only set up upper bound on the amount of allocated memory, and would just as likely have an adverse affect.

Answer 1

Disabling explicit GCs does not prevent buffers and thus the native memory they hold onto from being collected. But it may delay collections for a long time.

Which means memory allocated by direct buffers may accumulate for a long time before it is collected. In the long run that's not really a leak, but it will increase peak memory usage.

http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/4a1e42601d61/src/share/classes/java/nio/Bits.java

As I understand it that System.gc() call is there to free buffers when the reserveMemory limit is hit. After reserving the requested amount ByteBuffer.allocateDirect will call to Unsafe.allocateMemory which might do its own GC call, which should not be affected by DisableExplicitGC, if its attempt to mmap fails.

is there at least something we can do to reduce the frequency at which it's called?

It is only called when the MaxDirectMemorySize limitation is hit. If you can tune your GC or application code so that it meets one of the following options:

• it uses a fixed set of buffers (-> limitation will never be exceeded)
• buffers are collected early (shortlived buffers -> die in a young GC)
• the old generation gets collected regularly before direct buffer space runs out
• uses heap buffers instead of direct buffers

then System.gc() calls won't be necessary.

On hotspot there also exists a ExplicitGCInvokesConcurrent option. Maybe IBM's VM has something similar.