linux - fork() failing with Out of memory error

Question

The parent process fails with errno=12(Out of memory) when it tries to fork a child. The parent process runs on Linux 3.0 kernel - SLES 11. At the point of forking the child, the parent process has already used up around 70% of the RAM(180GB/256GB). Is there any workaround for this problem?

The application is written in C++, compiled with g++ 4.6.3.

Answer 1

Maybe virtual memory over commit is prevented in your system.

If it is prevented, then the virtual memory can not be bigger than sizeof physical RAM + swap. If it is allowed, then virtual memory can be bigger than RAM+swap.

When your process forks, your processes (parent and child) would have 2*180GB of virtual memory (that is too much if you don't have swap).

So, allow over commit by this way:

 echo 1 > /proc/sys/vm/overcommit_memory

It should help, if child process execves immediately, or frees allocated memory before the parent writes too much to own memory. So, be careful, out of memory killer may act if both processes keep using all the memory.

man page of proc(5) says:

/proc/sys/vm/overcommit_memory

This file contains the kernel virtual memory accounting mode. Values are: 0: heuristic overcommit (this is the default) 1: always overcommit, never check 2: always check, never overcommit

In mode 0, calls of mmap(2) with MAP_NORESERVE are not checked, and the default check is very weak, leading to the risk of getting a process "OOM-killed". Under Linux 2.4 any nonzero value implies mode 1. In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the size of the swap space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.

More information here: Overcommit Memory in SLES