Hi,
I are trying to build a Multi user chat application using Ejabberd to support upto 2000 users per room. I ran some tests and observed that the ejabberd server is not able to handle more than 400 users in a single MUC room.
At 400 users ejabberd is taking around 800 MB of RAM on the server. Also there are many message drops observed. I am using default ejabberd configuration. I also observed that we can have many more users (~2000) if we have multiple muc rooms with 30-40 users each.
I am wondering if there are any special confiugration parameters that we need to tweak ? or does ejabberd have trouble handling large number of users in a single room ?
Any advice on this is appreciated.
Meanwhile, i am have done my own reaseach and in my opinion, ejabberd is unable to handle the presence messages in a large muc, Looking at the XEP-405 standard, there should be an option to disable the presence broadcast to users. Does ejabberd support this feature ?
Here is a simple patch that
Here is a simple patch that disables presene broadcast. This means that all rooms are "dark": noone can see anybody else.
diff --git a/src/mod_muc/mod_muc_room.erl b/src/mod_muc/mod_muc_room.erl index 164805d..5513064 100644 --- a/src/mod_muc/mod_muc_room.erl +++ b/src/mod_muc/mod_muc_room.erl @@ -2096,6 +2096,8 @@ send_new_presence(NJID, Reason, StateData) -> Affiliation = get_affiliation(LJID, StateData), SAffiliation = affiliation_to_list(Affiliation), SRole = role_to_list(Role), + Destinations = [{LJID, Info} || {LJID, Info} <- + ?DICT:to_list(StateData#state.users), (NJID == Info#user.jid)], lists:foreach( fun({_LJID, Info}) -> ItemAttrs = @@ -2145,7 +2147,7 @@ send_new_presence(NJID, Reason, StateData) -> jlib:jid_replace_resource(StateData#state.jid, Nick), Info#user.jid, Packet) - end, ?DICT:to_list(StateData#state.users)). + end, Destinations). send_existing_presences(ToJID, StateData) -> @@ -2189,7 +2191,7 @@ send_existing_presences(ToJID, StateData) -> RealToJID, Packet) end - end, ?DICT:to_list(StateData#state.nicks)). + end, []). now_to_usec({MSec, Sec, USec}) ->Hi badlop, Thanks for the
Hi badlop,
Thanks for the patch and i am trying it out now. I wanted to know if joining >500 users into a single muc room is possible? was it desinged to handle such high numbers ? I am also curious about why ejabberd takes so much memory for higher numbers? Do we need to tweak any parameters on the ejabberd-muc configuration to acheive such numbers ? i have already played with all the params mentioned in ejabberd operation manual but dont see any improvements.
Any advice on this is highly appreciated.
Thanks
Hi badlop, I tested these
Hi badlop,
I tested these changes and the memory usage of ejabberd is very low (~ 160 mb ) for 1000 clients. This is very promising but ejabberd is crashing after some time. ejabberd crashes with the following error
=SUPERVISOR REPORT==== 30-Jan-2012::11:18:48 ===
Supervisor: {local,ejabberd_sup}
Context: shutdown_error
Reason: {{function_clause,
[{gen,call,
[{rex,
{error,{node_not_running,ejabberd@localhost}}},
'$gen_call',
{call,mnesia,remote_dirty_select,
[session,
[{{session,{'_','$1'},'_','_','_','_'},
[{'==',{node,'$1'},ejabberd@localhost}],
['$_']}]],
<0.39.0>},
infinity],
[{file,"gen.erl"},{line,146}]},
{gen_server,call,3,
[{file,"gen_server.erl"},{line,184}]},
{rpc,do_call,3,[{file,"rpc.erl"},{line,333}]},
{mnesia,do_dirty_rpc,5,
[{file,"mnesia.erl"},{line,1807}]},
{ejabberd_sm,'-recount_session_table/1-fun-2-',1,
[{file,"ejabberd_sm.erl"},{line,385}]},
{mnesia_tm,non_transaction,5,
[{file,"mnesia_tm.erl"},{line,734}]},
{ejabberd_sm,handle_info,2,
[{file,"ejabberd_sm.erl"},{line,324}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,597}]}]},
{gen_server,call,
[{rex,{error,{node_not_running,ejabberd@localhost}}},
{call,mnesia,remote_dirty_select,
[session,
[{{session,{'_','$1'},'_','_','_','_'},
[{'==',{node,'$1'},ejabberd@localhost}],
['$_']}]],
<0.39.0>},
infinity]}}
Offender: [{pid,<0.251.0>},
{name,ejabberd_sm},
{mfargs,{ejabberd_sm,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]
{"Kernel pid terminated",application_controller,"{application_terminated,mnesia,shutdown}"}
Crash dump was written to: //var/log/ejabberd/erl_crash_20120130-110341.dump
Kernel pid terminated (application_controller) ({application_terminated,mnesia,shutdown})
I have the entire crash log and if you want i can give you a link to download the crash log. I am trying to tweak the Mnesia tables setting and will let you know if that helps. In the mean time if you can provide some guidance on this, it will be much appreciated.
Thanks again for the patch.
With version 2.1.10, eajbberd
With version 2.1.10, eajbberd opens up a file for each connection made to the server. So ejabberd was crashing as soon as i reach 1000 users. I increased the number of open files to a higher number and then ejabberd stops crashing.
Is it a bug with version 2.1.10 ? Also can someone shed some light on what this open filed does ?
HI Badlop this patch solves
HI Badlop this patch solves the problem but i have some thoughts about how presence is handled in mod_muc module.
On a high level this is my understanding, For each user who joins the chat room, we are storing the presence of all the other users who are also present in the room. So as the number of clients increase the presence data associated with each client also increases exponentially. This eventually leads to the out of memory exceptions on the host.
Although storing the presence data for each user makes sense as we can support features like fine grained presence notifications for each user but at this point there is no way for the user to configure these in a MUC room.
If we can change the MUC module to store all the presence information at one place then i think we can avoid the excessive use of memory for a room which intends to server thousands of clients, on the flip side we will not be able to provide a fine grained presence settings for each user.
Can somebody confirm if my understanding of the presence problem is correct ? Also if someone can give me pointers on how the presence broadcast works and how we can tailor this module to suit the needs of large Chatrooms. it will be very helpful.