How to Deal With Non-heap or Native Memory Leak
when question occur out of heap
This week, I meet two very strange memory problems. Althought they are in different system, they have the same appearance —– Java process use much more memory over that JVM option setted.(e.g. Jvm option use 1.5G, but in
top command, java consume 2.9G memory).
direct buffer memory
First system was a data-transfer one. It uses Netty heavily. When get the Heap dump we see many DirectByteBuffer objects, it’s one of signals that we have direct buffer memory problem…then we use java.nio.Bits(see this) to confirm that direct buffer memory increase quickly when many request came.
At last, we found that netty sender product much requst that over receiver’s ability. So, we recheck our netty configuaration, and set
WATER_MARK option and check
channel.isWritable before write data to confirm that Netty has avalible to handle that request, if not we will try it later. Then direct buffer memory is under our control.
The other system was a Lib-scan one. It will scan many
.jar file every day.
At first, we try to dump heap many times, but we can not found any footmark in heap dump.
So, we change tool and use
pmap -x [pid] and
cat /proc/[pid]/smaps to see java memory usage.
First, we see find many big
anon blocks using 65500+ memory, like this..
00007f8b00000000 65512 40148 40148 rwx-- [ anon ] 00007f8b03ffa000 24 0 0 ----- [ anon ] 00007f8b04000000 65520 59816 59816 rwx-- [ anon ] 00007f8b07ffc000 16 0 0 ----- [ anon ]
Then we using
gdb to dump the content of these blocks
gdp -pid [pid] dump memory mem.bin 0x00007f8b00000000 0x00007f8b00000000+65512
Then let it visuable
cat mem.bin | strings
It’s suprise that the block content is
MANIFEST.MD file.. at this time, we start to suspect jar..
pmap -x [pid] | grep '.jar' | sort -k6 | uniq -c
After do some statistics, we found that many scanned jar file using memory more then once and never give back it…It’s the problem.
Then open JVM code —
ZipFile.c will found ZipFile/JarFile will use
mmap and only free then when
close() be called..
So, after recheck and fix unclosed
ZipFile/JarFile.. The problem was solved.
UPDATE: except un-close Jar file, before jdk1.7 URLClassLoader close JarFile lazy also lead mmap leak… we can use 1.7+ or hack it like this http://snipplr.com/view/24224/class-loader-which-close-opened-jar-files/ or try System.gc
UPDATE: the other accessible way may be let JarFile/ZipFile doesn’t use mmap…we can use jvm opt
-Dsun.zip.disableMemoryMapping to disable it.http://www.oracle.com/us/technologies/java/overview-156328.html
Using heap dump, we can find java heap problem, but Java and Java-Libary also have chance to make non-heap memory leak.
When use nio-libary, we must take care the direct memory usage, using nio libary carefully, and should better let
java.nio.Bits be monited(also this way is trick.)
In addition to
nio problem, mis-used JNI api also lead the non-heap memory leak, too. We should try
pmap or /proc/[pid]/smaps to found which jar/lib/stack/file might using much memory..then use
gdb to watch content of blocks to help our analysis.