linux - Elasticsearch process memory locking failed

Question

I have set boostrap.memory_lock=true Updated /etc/security/limits.conf added memlock unlimited for elastic search user

My elastic search was running fine for many months. Suddenly it failed 1 day back. In logs I can see below error and process never starts

ERROR: bootstrap checks failed memory locking requested for elasticsearch process but memory is not locked

I hit ulimit -as and I can see max locked memory set to unlimited. What is going wrong here? I have been trying for hours but all in vain. Please help.

OS is RHEL 7.2 Elasticsearch 5.1.2

ulimit -as output

core file size        (blocks -c) 0
data seg size         (kbytes -d) unlimited
scheduling policy            (-e) 0
file size            (blocks, -f) unlimited
pending signals              (-i) 83552
max locked memory    (kbytes, -l) unlimited
max memory size      (kbytes, -m) unlimited
open files                   (-n) 65536
pipe size         (512 bytes, -q) 8
POSIX message queues   (bytes,-q) 819200
real-time priority           (-r) 0
stack size            kbytes, -s) 8192
cpu time             seconds, -t) unlimited
max user processes           (-u) 4096
virtual memory       (kbytes, -v) unlimited
file locks                   (-x) unlimited

Answer 1

Here is what I have done to lock the memory on my ES nodes on RedHat/Centos 7 (it will work on other distributions if they use systemd).

You must make the change in 4 different places:

1) /etc/sysconfig/elasticsearch

On sysconfig: /etc/sysconfig/elasticsearch you should have:

ES_JAVA_OPTS="-Xms4g -Xmx4g" 
MAX_LOCKED_MEMORY=unlimited

(replace 4g with HALF your available RAM as recommended here)

2) /etc/security/limits.conf

On security limits config: /etc/security/limits.conf you should have

elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

3) /usr/lib/systemd/system/elasticsearch.service

On the service script: /usr/lib/systemd/system/elasticsearch.service you should uncomment:

LimitMEMLOCK=infinity

you should do systemctl daemon-reload after changing the service script

4) /etc/elasticsearch/elasticsearch.yml

On elasticsearch config finally: /etc/elasticsearch/elasticsearch.yml you should add:

bootstrap.memory_lock: true

Thats it, restart your node and the RAM will be locked, you should notice a major performance improvement.