Question regarding jemalloc and JVM
Vijay Parthasarathy
vparthasarathy at netflix.com
Mon Mar 26 09:51:54 PDT 2012
Hi Jason,
looks like it is problematic to use LD_PRELOAD i had various crashes. I understand it is too much to ask, any configuration to look out for?
I think most probably i will use JNI to load jemalloc (LD_LIBRARY_PATH), looks like that works fine so far.
grep -A2 Problematic *
hs_err_pid1309.log:# Problematic frame:
hs_err_pid1309.log-# C [libjemalloc.so+0xb79a]
hs_err_pid1309.log-#
--
hs_err_pid1622.log:# Problematic frame:
hs_err_pid1622.log-# C [libjemalloc.so+0x57f4] free+0x54
hs_err_pid1622.log-#
--
hs_err_pid16902.log:# Problematic frame:
hs_err_pid16902.log-# C [libjemalloc.so+0xb79a]
hs_err_pid16902.log-[error occurred during error reporting (printing problematic frame), id 0xb]
--
hs_err_pid16902.log:# Problematic frame:
hs_err_pid16902.log-# C [libjemalloc.so+0xb79a]
hs_err_pid16902.log-[error occurred during error reporting (printing problematic frame), id 0xb]
--
hs_err_pid2813.log:# Problematic frame:
hs_err_pid2813.log-# C [libjemalloc.so+0xb79a]
hs_err_pid2813.log-[error occurred during error reporting (printing problematic frame), id 0xb]
--
hs_err_pid2813.log:# Problematic frame:
hs_err_pid2813.log-# C [libjemalloc.so+0xb79a]
hs_err_pid2813.log-[error occurred during error reporting (printing problematic frame), id 0xb]
--
hs_err_pid29892.log:# Problematic frame:
hs_err_pid29892.log-# C [libjemalloc.so+0xb79a]
hs_err_pid29892.log-#
--
hs_err_pid30273.log:# Problematic frame:
hs_err_pid30273.log-# C [libjemalloc.so+0xb79a]
hs_err_pid30273.log-#
--
hs_err_pid30645.log:# Problematic frame:
hs_err_pid30645.log-# C [libjemalloc.so+0xb79a]
hs_err_pid30645.log-#
--
hs_err_pid4037.log:# Problematic frame:
hs_err_pid4037.log-# C [libjemalloc.so+0xb79a]
hs_err_pid4037.log-[error occurred during error reporting (printing problematic frame), id 0xb]
--
hs_err_pid7733.log:# Problematic frame:
hs_err_pid7733.log-# C [libc.so.6+0x618a2]
hs_err_pid7733.log-[error occurred during error reporting (printing problematic frame), id 0xb]
--
hs_err_pid7733.log:# Problematic frame:
hs_err_pid7733.log-# C [libc.so.6+0x618a2]
hs_err_pid7733.log-[error occurred during error reporting (printing problematic frame), id 0xb]
[vijay_tcasstest at vijay_tcass-i-a91ee8cd crash]$
On Mar 25, 2012, at 12:21 AM, VIJAY2WIN at gmail.com<mailto:VIJAY2WIN at gmail.com> wrote:
Thanks Jason,
I had one crash in the with in JVM version 1.6 and it crashed with following... So far with JVM 1.7 the test looks good still waiting for a way to crash it (I did verify that it uses JEMalloc).
# Problematic frame:
# C [libjemalloc.so+0xb79a]
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
--------------- T H R E A D ---------------
Current thread (0x00002aac4ede7000): JavaThread "Selector-Thread-3" [_thread_new, id=1555, stack(0x0000000000000000,0x0000000000000000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00002ab252cd9368
Registers:
RAX=0x00000000c217906f, RBX=0x00002aac42110000, RCX=0x0000000000000002, RDX=0x0000000000000000
RSP=0x0000000042aa3eb0, RBP=0x0000000000000020, RSI=0x00002aac42110040, RDI=0x0000000000000006
R8 =0x00002aaaab0009c0, R9 =0x0000000000000003, R10=0x00002aac42110ff0, R11=0x00000000c217906f
R12=0xffffffffffffffe8, R13=0x0000000042aa3f80, R14=0x0000000000000020, R15=0x0000000000000000
RIP=0x00002aaaaaab879a, EFL=0x0000000000010283, CSGSFS=0x000000000000e033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x0000000042aa3eb0)
0x0000000042aa3eb0: 0000000000000000 0000000000000020
0x0000000042aa3ec0: 0000000042aa3fe0 0000000042aa3ff0
0x0000000042aa3ed0: 0000000042aa3f80 0000000000000020
0x0000000042aa3ee0: 0000000000000000 00002aaaaaab2740
0x0000000042aa3ef0: 0000000042aa4940 0000003a0cc08373
0x0000000042aa3f00: 0000000000000000 0000000000000000
0x0000000042aa3f10: 0000000000000000 0000000000000000
0x0000000042aa3f20: 0000000000000000 0000000000000000
0x0000000042aa3f30: 00002aad67d6c728 0000000000000000
0x0000000042aa3f40: 0000003a0cc0a0e7 0000000042aa3ff8
0x0000000042aa3f50: 0000000042aa3fe0 0000000042aa3ff0
0x0000000042aa3f60: 0000000042aa3f80 00002aac4ede7000
0x0000000042aa3f70: 0000000000000000 00002aaaabdaa354
0x0000000042aa3f80: 0000000000000000 0000000000000009
0x0000000042aa3f90: 0000000000000000 0000000042aa5000
0x0000000042aa3fa0: 0000000000021000 00002aaaac432600
0x0000000042aa3fb0: 00002aaaabb68da0 0000000000000007
0x0000000042aa3fc0: 0000000042aa3ff0 00002aac4ede7000
0x0000000042aa3fd0: 00002aac4ee12400 00002aac4ee06400
0x0000000042aa3fe0: 0000000042aa4000 00002aaaabdaa415
0x0000000042aa3ff0: 0000000042aa4020 00002aaaabea046e
0x0000000042aa4000: 0000000042aa4020 00002aaaabe967b1
0x0000000042aa4010: 00002aac4ee12400 00002aac4ede7000
0x0000000042aa4020: 0000000042aa4040 00002aaaabe9cb14
0x0000000042aa4030: 00002aac4ede7000 0000000000000000
0x0000000042aa4040: 0000000042aa4110 00002aaaabda8ed4
0x0000000042aa4050: 0000000000000000 0000000000000000
0x0000000042aa4060: 0000000000000000 0000000000000000
0x0000000042aa4070: 0000000000000000 0000000000000000
0x0000000042aa4080: 0000000000000000 0000000000000000
0x0000000042aa4090: 0000000000000000 0000000000000000
0x0000000042aa40a0: 0000000000000000 0000000000000000
Instructions: (pc=0x00002aaaaaab879a)
0x00002aaaaaab878a: 06 89 46 08 7d 02 89 06 44 8b 5e 08 4c 8b 56 10
0x00002aaaaaab879a: 4f 8b 34 da 4d 85 f6 0f 84 71 02 00 00 45 84 ff
Stack: [0x0000000000000000,0x0000000000000000], sp=0x0000000042aa3eb0, free space=1092239k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libjemalloc.so+0xb79a]
[error occurred during error reporting (printing native stack), id 0xb]
On Mar 24, 2012, at 9:00 PM, Jason Evans wrote:
On Mar 24, 2012, at 11:46 AM, Jason Evans wrote:
On Mar 23, 2012, at 4:21 PM, Vijay wrote:
Is there any reason i should not be using LD_PRELOAD to force JVM use jemalloc instead of native gcc malloc?
Background: This is for Apache-Cassandra we do off-heap cache where we explicitly call malloc and free and by using jemalloc we see a fair amount of space savings and was wondering if there is anything which we have to consider before using it.
LD_PRELOAD'ing jemalloc should be okay as long as the JVM doesn't statically link a different malloc implementation. I expect that if it isn't safe, you'll experience crashes quite early on, so give it a try and see what happens.
I was just looking through some of my notes and noticed an issue that may be relevant here. jemalloc has a configure option called lazy-lock that has apparently caused problems in some Java use cases. Java was using dlopen() to load a library that was dynamically linked with jemalloc, so jemalloc came along too late to intercept pthread_create() calls and turn on locking. I don't think your use case will hit the same issue, but beware of dlopen() anywhere in the mix.
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jemalloc.net/mailman/jemalloc-discuss/attachments/20120326/d0450b79/attachment.html>
More information about the jemalloc-discuss
mailing list