out of memory - Java: Unable to create new native thread

Question

I have a Java application that is hosted on by a web hosting company. Every few days my app goes down with:

[2011-03-09 15:52:14,501] ERROR http-12021-9 
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:597)

The hosting company says it means my app is leaking memory, but the tools I have are showing free memory is still available. Since the error is always creating a new native thread my thinking is that the issue is in the JVM config/OS resources.

How do I prevent this error from happening?

Answer 1

One possibility is that you have reached your user limit for the number of open files.

I believe that every Process/Thread consumes one or more file descriptors.

For example, when this occurs for your user then "no" shell command will work, since shell commands fork off a process to execute (you see errors like "-bash: fork: retry: Resource temporarily unavailable")

I hit this issue and found that only the current user was unable to spawn procs... other users were uneffected.

To resolve, up your ulimit -n (max files open) setting... details follow.

You can see your user limits with the command:

ulimit -a

Up your max file limit with the following:

ulimit -n 65536 

Here is what I have right now:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256797
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 75000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To see all the explicit limits for your system:

cat /etc/security/limits.conf

Please note: I'm using Oracle Linux 6.3 - results may differ slightly between distros.