php - Memcached: How to break the limitation when retrieving all keys

Question

Due to the fact I could not find any reliable program in the net which retrieves all memcache keys once the key count in a slab > 500k I wanted to write my own program in PHP.

Then I realized there is a reason why all of the programs did not really work for this high amount of keys.

When I read the keys with stats cachedump <slab-id> <limit> it returns only a certain amount of keys which ends - in my case - by around 30k.

How can I get at least these 500k which may be in one slab?

All the "posts" and "answers" which suggest to use memdump do not work. They have this limitation as well.

I am using memcached 1.4.25

Answer 1

After debugging the memcached source I realized the limitation is in items.c/ item_cachedump() caused by this line:

unsigned int memlimit = 2 * 1024 * 1024;   /* 2MB max response size */

Due to this assignment the allocated buffer limits the amount of the returned keys because the total length of all keys [including after each key name] must not exceed this size.

I could solve for my case the problem in that way that I changed it to:

unsigned int memlimit = 128 * 1024 * 1024;

and recompiling memcached. Now I can read around 700k keys [I did not have more keys to max the function].

I found out that a very high value makes the system instable. With a value of 500 * 1024 * 1024 [e.q. 500MB] my system almost crashed with this messages:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

Even my server has 5GB RAM I don't really understand why but in the end a value of 128MB is a usable value.

I asked the memcached developer team to add a switch which can set the value in the command line.