Hello,
While playing around with XMPP libraries for different languages, I've noticed that I can make my ejabberd instance (respectively the beam process) crash instantly when with a client that uses the Ruby XMPP4R library IF the listener is TLS enabled:
[{5222, ejabberd_c2s, [{access, c2s}, {shaper, c2s_shaper},
{max_stanza_size, 65536},
starttls, {certfile, "./ssl.pem"}]},
These are just the unmodified examples that come with XMPP4R. They work when I remove the TLS stuff from the listener. Also, I have working examples using the Perl Net::Jabber library which work for listeners using either starttls, starttls_required or tls, so there doesn't seem to be anything wrong with SSL support per se in my Erlang environment.
This is ejabberd 1.1.2 using OTP R11B-0 (or so it says) on Gentoo Linux.
But as soon as I connect with the Ruby thing, the server REALLY crashes and burns:
=INFO REPORT==== 12-Oct-2006::17:00:46 ===
I(<0.228.0>:ejabberd_listener:90): (#Port<0.335>) Accepted connection {{127,0,0,1},55255} -> {{127,0,0,1},5222}
D(<0.295.0>:ejabberd_receiver:261) : Received XML on stream = "<stream:stream xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client' to='localhost' xml:lang='en' version='1.0' >"
D(<0.294.0>:ejabberd_c2s:1207) : Send XML on stream = "<?xml version='1.0'?><stream:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' id='2549000267' from='localhost' version='1.0' xml:lang='en'>"
D(<0.294.0>:ejabberd_c2s:1207) : Send XML on stream = "<stream:features><starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/><mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><mechanism>DIGEST-MD5</mechanism><mechanism>PLAIN</mechanism></mechanisms><register xmlns='http://jabber.org/features/iq-register'/></stream:features>"
D(<0.295.0>:ejabberd_receiver:261) : Received XML on stream = "<starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>"
D(<0.295.0>:ejabberd_receiver:261) : Received XML on stream = []
D(<0.294.0>:ejabberd_c2s:1207) : Send XML on stream = "<proceed xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>"
D(<0.295.0>:ejabberd_receiver:261) : Received XML on stream = []
/usr/bin/ejabberd: line 12: 2840 Segmentation fault erl -pa /usr/lib/erlang/lib/ejabberd-1.1.2/ebin -sname ejabberd -s ejabberd -ejabberd config \"/etc/jabber/ejabberd.cfg\" log_path \"/var/log/jabber/ejabberd.log\" -kernel inetrc \"/etc/jabber/inetrc\" -sasl sasl_error_logger \{file,\"/var/log/jabber/sasl.log\"\} -mnesia dir \"/var/spool/jabber\" $@
It might as well be that XMPP4R is doing something wrong, but whatever it is shouldn't crash the server. It may also be something in my environment. I've recompiled both Erlang/OTP and ejabberd, to no avail. And if my library chain is botched, why's it working with Perl/Net::Jabber?
Any ideas?
thanks
-stefan
After some more monkeying
After some more monkeying around I've noticed a different manifestation of the problem where glibc catches an invalid pointer and provides a backtrace. For different runs of the same problem I get mostly the segfault version and sometimes this. Something is well and truly botched.
=INFO REPORT==== 12-Oct-2006::22:25:37 ===
I(<0.210.0>:ejabberd_listener:90): (#Port<0.317>) Accepted connection
{{127,0,0,1},53263} -> {{127,0,0,1},5222}
D(<0.277.0>:ejabberd_receiver:261) : Received XML on stream = ""
D(<0.276.0>:ejabberd_c2s:1207) : Send XML on stream = "<?xml
version='1.0'?>"
D(<0.276.0>:ejabberd_c2s:1207) : Send XML on stream =
"DIGEST-MD5PLAIN"
D(<0.277.0>:ejabberd_receiver:261) : Received XML on stream = ""
D(<0.277.0>:ejabberd_receiver:261) : Received XML on stream = []
D(<0.276.0>:ejabberd_c2s:1207) : Send XML on stream = ""
D(<0.277.0>:ejabberd_receiver:261) : Received XML on stream = []
*** glibc detected *** /usr/lib/erlang/erts-5.5/bin/beam: free(): invalid
pointer: 0x081748c0 ***
=======
Backtrace: =========
/lib/libc.so.6[0xb7e1ab9b]
/lib/libc.so.6(__libc_free+0x79)[0xb7e1c0d7]
/usr/lib/erlang/erts-5.5/bin/beam[0x806861b]
/lib/libz.so(inflateEnd+0x26)[0xb7f5dce1]
/usr/lib/libcrypto.so.0.9.8[0xb6c4cdef]
/usr/lib/libcrypto.so.0.9.8[0xb6bc3f92]
/usr/lib/libcrypto.so.0.9.8(CRYPTO_free_ex_data+0x2f)[0xb6bc3c17]
/usr/lib/libcrypto.so.0.9.8[0xb6c4cc60]
/usr/lib/libcrypto.so.0.9.8(COMP_CTX_free+0x23)[0xb6c4c8d3]
/usr/lib/libssl.so.0.9.8(ssl_clear_cipher_ctx+0x68)[0xb6762c54]
/usr/lib/libssl.so.0.9.8(SSL_free+0xf5)[0xb6764452]
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/tls_drv.so[0xb7f677f6]
/usr/lib/erlang/erts-5.5/bin/beam[0x8089882]
/usr/lib/erlang/erts-5.5/bin/beam(erts_do_exit_port+0x157)[0x808aa79]
/usr/lib/erlang/erts-5.5/bin/beam(port_close_1+0x31)[0x80b43e9]
/usr/lib/erlang/erts-5.5/bin/beam(process_main+0x39ee)[0x80bcb90]
/usr/lib/erlang/erts-5.5/bin/beam(erl_start+0x854)[0x80731d3]
/usr/lib/erlang/erts-5.5/bin/beam(main+0xe)[0x8067cce]
/lib/libc.so.6(__libc_start_main+0xdc)[0xb7dcf864]
/usr/lib/erlang/erts-5.5/bin/beam(sinh+0x7d)[0x8067c41]
======= Memory map: ========
08048000-08103000
r-xp 00000000 08:13 4948676 /usr/lib/erlang/erts-5.5/bin/beam
08103000-0812c000
rwxp 000ba000 08:13 4948676 /usr/lib/erlang/erts-5.5/bin/beam
0812c000-081d9000
rwxp 0812c000 00:00 0 [heap]
b6600000-b6608000
rwxp b6600000 00:00 0
b6608000-b6700000
---p b6608000 00:00 0
b671d000-b6726000 r-xp 00000000 08:13
4604611 /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libgcc_s.so.1
b6726000-b6727000
rwxp 00008000 08:13 4604611
/usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libgcc_s.so.1
b6727000-b6740000
r-xp 00000000 08:13 4885461 /usr/lib/libexpat.so.1.5.0
b6740000-b6742000
rwxp 00019000 08:13 4885461 /usr/lib/libexpat.so.1.5.0
b6742000-b6771000
r-xp 00000000 08:13 4869504 /usr/lib/libssl.so.0.9.8
b6771000-b6775000
rwxp 0002e000 08:13 4869504 /usr/lib/libssl.so.0.9.8
b6792000-b67bc000 r-xp
00000000 08:13 4767804
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/stringprep_drv.so
b67bc000-b67bd000
rwxp 0002a000 08:13 4767804
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/stringprep_drv.so
b6924000-b6b8c000
rwxp b6924000 00:00 0
b6b8c000-b6c73000
r-xp 00000000 08:13 4869480 /usr/lib/libcrypto.so.0.9.8
b6c73000-b6c88000
rwxp 000e6000 08:13 4869480 /usr/lib/libcrypto.so.0.9.8
b6c88000-b6c8b000
rwxp b6c88000 00:00 0
b6c8b000-b6c8d000 r-xp 00000000 08:13 5374118
/usr/lib/erlang/lib/crypto-1.5/priv/lib/crypto_drv.so
b6c8d000-b6c8e000
rwxp 00001000 08:13 5374118
/usr/lib/erlang/lib/crypto-1.5/priv/lib/crypto_drv.so
b6c8e000-b7291000
rwxp b6c8e000 00:00 0
b7291000-b7292000
---p b7291000 00:00 0
b7292000-b7dba000
rwxp b7292000 00:00 0
b7dba000-b7eca000 r-xp 00000000 08:13 4575233
/lib/libc-2.4.so
b7eca000-b7ecc000
r-xp 0010f000 08:13 4575233 /lib/libc-2.4.so
b7ecc000-b7ece000
rwxp 00111000 08:13 4575233 /lib/libc-2.4.so
b7ece000-b7ed1000 rwxp
b7ece000 00:00 0
b7ed1000-b7ed8000
r-xp 00000000 08:13 4575209 /lib/librt-2.4.so
b7ed8000-b7eda000 rwxp 00006000 08:13 4575209 /lib/librt-2.4.so
b7eda000-b7edb000
rwxp b7eda000 00:00 0
b7edb000-b7eea000
r-xp 00000000 08:13 4575152 /lib/libpthread-2.4.so
b7eea000-b7eeb000 r-xp 0000e000 08:13 4575152
/lib/libpthread-2.4.so
b7eeb000-b7eec000
rwxp 0000f000 08:13 4575152 /lib/libpthread-2.4.so
b7eec000-b7eee000
rwxp b7eec000 00:00 0
b7eee000-b7f1b000 r-xp 00000000 08:13
4884778 /lib/libncurses.so.5.5
b7f1b000-b7f23000
rwxp 0002c000 08:13 4884778 /lib/libncurses.so.5.5
b7f23000-b7f24000 rwxp b7f23000 00:00 0
b7f24000-b7f46000
r-xp 00000000 08:13 4575148 /lib/libm-2.4.so
b7f46000-b7f48000
rwxp 00021000 08:13 4575148 /lib/libm-2.4.so
b7f48000-b7f4a000 r-xp 00000000 08:13 4575149
/lib/libdl-2.4.so
b7f4a000-b7f4c000
rwxp 00001000 08:13 4575149 /lib/libdl-2.4.so
b7f51000-b7f52000
r-xp 00000000 08:13 4767802
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/iconv_erl.so
b7f52000-b7f53000
rwxp 00001000 08:13 4767802
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/iconv_erl.so
b7f53000-b7f56000 r-xp
00000000 08:13 4767792
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/expat_erl.so
b7f56000-b7f57000
rwxp 00003000 08:13 4767792
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/expat_erl.so
b7f57000-b7f65000
r-xp 00000000 08:13 4493993 /lib/libz.so.1.2.3
b7f65000-b7f66000 rwxp 0000d000 08:13 4493993 /lib/libz.so.1.2.3
b7f66000-b7f68000
r-xp 00000000 08:13 4767809
/usr/lib/erlang/lib/ejabberd-1.1.2/priv/lib/tls_drv.so
b7f68000-b7f69000 rwxp 00001000 08:13/usr/bin/ejabberd: line 12:
14967 Aborted erl -pa /usr/lib/erlang/lib/ejabberd-1.1.2/ebin
-sname ejabberd -s ejabberd -ejabberd config \"/etc/jabber/ejabberd.cfg\"
log_path \"/var/log/jabber/ejabberd.log\" -kernel inetrc
\"/etc/jabber/inetrc\" -sasl sasl_error_logger
\{file,\"/var/log/jabber/sasl.log\"\} -mnesia dir \"/var/spool/jabber\" $@
Just for good measure, I
Just for good measure, I also tried with Erlang/OTP R11B-1 and Erlang/OTP R10B-10. Makes no difference.
Same problem here
I have this problem too on another ejabberd-node. Any debugging information I can provide?
Do you use Gentoo too?
I have this problem too on another ejabberd-node. Any debugging information I can provide?
Do you use Gentoo, too? Which version of: ejabberd, erlang, xmpp4r, openssl libraries?
A work-around for XMPP4R
I'm having the same problem for weeks now (server is Gentoo, too), sorry for not reporting.
If the XMPP4R client is your own, try the following on a Jabber::Client instance:
client.allow_tls = false
Obviously ejabberd should be fixed, but you may ping me at xmpp:astro@spaceboyz.net for any bugs you find in XMPP4R's (not very complex) TLS code.
Dirty solution.
I have the same problem on Gentoo linux.
- ejabberd-1.1.2
- openssl-0.9.7, openssl-0.9.8
- almost any version of erlang from portage
- zlib version 1.2.3 (or -r1, doesn't matter)
After playing with gdb I've noticed, that bug appears in zlib module.
When I emerge openssl without zlib use flag, this bug vanishes.
I think this is ejabberd tls_drv.so bug, because other applications that use openssl work fine with stream compression.
thanks
I had the same problems, but also with S2S TLS connections (as posted on the ejabberd mailinglist). I disabled zlib support in openssl, now TLS works and doesn't kill the server daemon.(but it's surely not a nice solution)