ejabberd - Comments for "Can't Access Virtual Hosts Page"

LDAP w/ lots of users, solution

msmith — Tue, 22 Nov 2005 19:58:42 +0000

Hi,

This problem has been resolved:

http://www.jabber.ru/bugzilla/show_bug.cgi?id=179

I'm not sure if this will be merged with the CVS branch or what, but its just a one line fix, well, and changing CMD_TIMEOUT to something larger than 5 seconds.

Thanks for everyones help!

--Marc

erlang CMD_TIMEOUT limit?

msmith — Thu, 17 Nov 2005 18:48:07 +0000

This couldn't be caused by another variable set in somewhere is the erlang install, could it?

I can adjust CMD_TIMEOUT to something lower (4000, 3000, etc.) and the time will decrease according to what its set at. I can't get the time to last any longer than 5 seconds and some change. I've tried setting CMD_TIMEOUT to 5050 and all the way up to 100000. It never goes past ~6 seconds!

esdev2# time su ejabberd -c "/usr/local/bin/ejabberdctl ejabberd@`hostname -s` registered-users"
Can't get list of registered users at node ejabberd@esdev2: {'EXIT',
                                                                {timeout,
                                                                    {gen_fsm,
                                                                        sync_send_event,
                                                                        ['eldap_ejabberd_esdev2.mcc.edu',
                                                                         {search,
                                                                             {eldap_search,
                                                                                 wholeSubtree,
                                                                                 "ou=Users,dc=mcc,dc=edu",
                                                                                 {present,
                                                                                     "uid"},
                                                                                 ["uid"],
                                                                                 false,
                                                                                 0}}]}}}
0.025u 0.520s 0:05.79 9.3%      580+5337k 0+0io 0pf+0w
esdev2#

This test server is a VMware virtual machine -- it couldn't be anything weird with time (the VMs are always a bit funk with the time stuff)?

Does anyone else use ejabberd & LDAP with lots (+13,000) users?

Thanks,

Marc

Won't Patch Correctly

msmith — Mon, 14 Nov 2005 21:33:36 +0000

Hi,

Thanks for the reply, but I can't get eldap.erl to patch correctly:

esdev2# patch < /root/ejabberd/eldap.erl.patch
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- eldap.erl (revisiÃ³n: 433)
|+++ eldap.erl (copia de trabajo)
--------------------------
Patching file eldap.erl using Plan A...
Hunk #1 failed at 559.
Hunk #2 succeeded at 732 (offset -1 lines).
Hunk #3 succeeded at 765 (offset -1 lines).
1 out of 3 hunks failed--saving rejects to eldap.erl.rej
done

esdev2# cat eldap.erl.rej
***************
*** 559,565 ****
log2("~p~n",[{Name, Request}], S),
{ok, Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message),
ok = gen_tcp:send(S#eldap.fd, Bytes),
- Timer = erlang:start_timer(?CMD_TIMEOUT, self(), {cmd_timeout, Id}),
New_dict = dict:store(Id, [{Timer, From, Name}], S#eldap.dict),
{ok, S#eldap{id = Id,
dict = New_dict}}.
--- 559,567 ----
log2("~p~n",[{Name, Request}], S),
{ok, Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message),
ok = gen_tcp:send(S#eldap.fd, Bytes),
+ Timer = erlang:start_timer(43000, self(), {cmd_timeout, Id}),
+ io:format(" -- eldap - start_timer - ~p --~ncommand: ~p~nfrom: ~p~ns:~p~ntimer: ~p~n~n",
+ [erlang:now(), Command, From, S, Timer]),
New_dict = dict:store(Id, [{Timer, From, Name}], S#eldap.dict),
{ok, S#eldap{id = Id,
dict = New_dict}}.

esdev2# cat eldap.erl.patch
--- eldap.erl (revisiÃ³n: 433)
+++ eldap.erl (copia de trabajo)
@@ -559,7 +559,9 @@
log2("~p~n",[{Name, Request}], S),
{ok, Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message),
ok = gen_tcp:send(S#eldap.fd, Bytes),
- Timer = erlang:start_timer(?CMD_TIMEOUT, self(), {cmd_timeout, Id}),
+ Timer = erlang:start_timer(43000, self(), {cmd_timeout, Id}),
+ io:format(" -- eldap - start_timer - ~p --~ncommand: ~p~nfrom: ~p~ns:~p~ntimer: ~p~n~n",
+ [erlang:now(), Command, From, S, Timer]),
New_dict = dict:store(Id, [{Timer, From, Name}], S#eldap.dict),
{ok, S#eldap{id = Id,
dict = New_dict}}.
@@ -732,6 +733,7 @@

cancel_timer(Timer) ->
erlang:cancel_timer(Timer),
+ io:format(" -- eldap - cancel_timer - ~p --~ntimer: ~p~n~n", [erlang:now(), Timer]),
receive
{timeout, Timer, _} ->
ok
@@ -764,6 +766,8 @@
%% Sort out timed out commands
%%-----------------------------------------------------------------------
cmd_timeout(Timer, Id, S) ->
+ io:format(" -- eldap - cmd_timeout - ~p --~ntimer: ~p~nid: ~p~ns:~p~n~n",
+ [erlang:now(), Timer, Id, S]),
Dict = S#eldap.dict,
case dict:find(Id, Dict) of
{ok, [{Timer, From, Name}|Res]} ->

I thought maybe this was because I wasn't using the CVS version, so I got eldap.erl from CVS, but it failed with the same message.

Thanks,

Marc

some help for debugging

mfoss — Mon, 14 Nov 2005 20:29:34 +0000

Offtopic: There's a new feature on this forum: now you can see the thread plain instead of nested.

In topic: look at ejabberd/src/eldap/eldap.erl line 554:

send_command(Command, From, S) ->
...
    {ok, Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message),
    ok = gen_tcp:send(S#eldap.fd, Bytes),
    Timer = erlang:start_timer(?CMD_TIMEOUT, self(), {cmd_timeout, Id}),
...

That function seams to be the one that sends the request and starts the timer. Check the documentation for start_timer. It only says: 'Time is a non-negative integer... Time ms ... The timeout value must fit in 32 bits.'

To help your debugging, you can apply this patch to eldap.erl (note that I've not even tried to compile this):

--- eldap.erl	(revisiÃ³n: 433)
+++ eldap.erl	(copia de trabajo)
@@ -559,7 +559,9 @@
     log2("~p~n",[{Name, Request}], S),
     {ok, Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message),
     ok = gen_tcp:send(S#eldap.fd, Bytes),
-     Timer = erlang:start_timer(?CMD_TIMEOUT, self(), {cmd_timeout, Id}),
+     Timer = erlang:start_timer(43000, self(), {cmd_timeout, Id}),
+     io:format(" -- eldap - start_timer - ~p --~ncommand: ~p~nfrom: ~p~ns:~p~ntimer: ~p~n~n", 
+          [erlang:now(), Command, From, S, Timer]),
     New_dict = dict:store(Id, [{Timer, From, Name}], S#eldap.dict),
     {ok, S#eldap{id = Id,
 		 dict = New_dict}}.
@@ -732,6 +733,7 @@
 
 cancel_timer(Timer) ->
     erlang:cancel_timer(Timer),
+     io:format(" -- eldap - cancel_timer - ~p --~ntimer: ~p~n~n", [erlang:now(), Timer]),
     receive
 	{timeout, Timer, _} ->
 	    ok
@@ -764,6 +766,8 @@
 %% Sort out timed out commands
 %%-----------------------------------------------------------------------
 cmd_timeout(Timer, Id, S) ->
+     io:format(" -- eldap - cmd_timeout - ~p --~ntimer: ~p~nid: ~p~ns:~p~n~n", 
+          [erlang:now(), Timer, Id, S]),
     Dict = S#eldap.dict,
     case dict:find(Id, Dict) of
 	{ok, [{Timer, From, Name}|Res]} ->

It forces a timeout of 43 seconds for commands, and prints on the log file some messages.

Anyway, I'm thinking now that maybe ejabberd should not list all the users on the LDAP server, but only the ones that have sometime logged on ejabberd. This behaviour should be used on Shared Roster Groups too: when you specify @all@, instead of adding all LDAP users to a shared roster group, add only the ones that logged on the Jabber server.

Does TIMEOUT variable have a ceiling?

msmith — Mon, 14 Nov 2005 18:57:04 +0000

Hi,

I really don't know that much about programming, but I have done some experiments with this problem, and I can't make the timeout any longer than 6 seconds (real time).

-define(RETRY_TIMEOUT, 5000).
-define(BIND_TIMEOUT, 10000).
-define(CMD_TIMEOUT, 5000).

Those are the default values, but if I set each to say 100,000,000 the timeout is still only 6 seconds. I check this by doing a 'time ejabberdctl ejabberd@host registered-users' and its always 6 no matter how high I make it. Its also only 6 seconds with the default values.

Now, if I lower these values, say half the default values, it times out in half the time, 3 seconds. I've been changing all 3, but I'm assuming the only one I really need to be changing for my problem is CMD_TIMEOUT? Or maybe I need to change the others too?

So setting the values lower changed the results with my problem. I just can't get the timeout to increase past 6 seconds. So, I guess I'm just wondering if the variable is the wrong type or something similar. It seems as though I've hit some kind of ceiling.

Thanks for any help!

--Marc

Advanced LDAP config in ejabberd.cfg?

msmith — Fri, 11 Nov 2005 18:53:09 +0000

Hi,

I don't see any errors in my slapd log file when trying to access the virtual hosts page.

Nov 11 11:17:45 acad1 slapd[11452]: conn=4 op=34 SRCH base="ou=Users,dc=mcc,dc=edu" scope=2 deref=0 filter="(uid=msmith)"
Nov 11 11:17:45 acad1 slapd[11452]: conn=4 op=34 SEARCH RESULT tag=101 err=0 nentries=1 text=
Nov 11 11:17:45 acad1 slapd[11456]: conn=3 op=27 BIND anonymous mech=implicit ssf=0
Nov 11 11:17:45 acad1 slapd[11456]: conn=3 op=27 BIND dn="uid=msmith,ou=Users,dc=mcc,dc=edu" method=128
Nov 11 11:17:45 acad1 slapd[11456]: conn=3 op=27 BIND dn="uid=msmith,ou=Users,dc=mcc,dc=edu" mech=SIMPLE ssf=0
Nov 11 11:17:45 acad1 slapd[11456]: conn=3 op=27 RESULT tag=97 err=0 text=
Nov 11 11:17:45 acad1 slapd[11452]: conn=4 op=35 SRCH base="ou=Users,dc=mcc,dc=edu" scope=2 deref=0 filter="(uid=*)"
Nov 11 11:17:45 acad1 slapd[11452]: conn=4 op=35 SRCH attr=uid
Nov 11 11:17:51 acad1 slapd[11452]: conn=4 op=35 SEARCH RESULT tag=101 err=0 nentries=14055 text=

Other things that use LDAP work fine too... doing a 'getent passwd' with nss_ldap works fine, but I'm sure different applications can work differently and its not always the same. Are there any other settings for ejabberd with LDAP? Like advanced search filters?

I already have one setting in my slapd.conf for searches:

sizelimit 65435

I'll take a look for some other slapd timeout type settings. Should I turn up slapd logging level? Maybe I'm not seeing everything with a '256' loglevel?

Thanks for everyones help.

--Marc

Check your LDAP server logs and settings

gandrews — Fri, 11 Nov 2005 06:43:19 +0000

Check the logfiles and settings in your LDAP server. LDAP servers can limit the number of items returned in a search (or the amount of time a search is allowed to take).

Perhaps the limit you are reaching is configured in your LDAP server instead of your ejabberd.

Greg

Works fine with ~100 users

msmith — Thu, 10 Nov 2005 22:04:06 +0000

I cloned my LDAP server and added roughly 100 users, and it works fine. So, 13,000 users is just too much for it? Is there anywhere else a 'timeout' variable is set besides what I already tried?

Has anyone else been using ejabberd & LDAP with a large number of users (13,000 or more)?

Thanks,

Marc

Doesn't seem to help

msmith — Thu, 10 Nov 2005 13:36:18 +0000

I can't get ejabberd to compile with out the FreeBSD ports system, so I just used that:

# cd /usr/ports/net/ejabberd
# make clean
# make configure
# vi work/ejabberd-0.9.8/src/eldap/eldap.erl

I changed the 3 'TIMEOUT' to look like this:

-define(RETRY_TIMEOUT, 5000000).
-define(BIND_TIMEOUT, 10000000).
-define(CMD_TIMEOUT, 5000000).

# make build
# make install

I fired up ejabberd and tail'd the sasl.log file -- I get the same problem, but it doesn't seem to take any longer to appear after clicking something that trys to pull a list of users. Which I would think I'd notice a difference when multiplying the values by 1000.

I'll try another experiment -- I'll clone my LDAP server and take out all users except for 10 and see how that works.

I'll post again soon.

Thanks for the help.

--Marc

Maybe it can't handle all

mfoss — Thu, 10 Nov 2005 10:10:24 +0000

Maybe it can't handle all the users in my LDAP tree (~13,000)?

Looking at the error message, it seems possible:

    error_info: {timeout,{gen_fsm,sync_send_event,
                                   ['eldap_ejabberd_chatty.mcc.edu',
                                    {search,
                                        {eldap_search,
                                            wholeSubtree,
                                            "dc=mcc,dc=edu",
                                            {present,"uid"},
                                            ["uid"],
                                            false,
                                            0}}]}}

With so many users, it could be the default timeout happens without receiving the answer.

If you are still interested in that list of users, you can try to increment those constants defined in ejabberd/src/eldap/eldap.erl (maybe they are related to the timeout you experience):

-define(RETRY_TIMEOUT, 5000).
-define(BIND_TIMEOUT, 10000).
-define(CMD_TIMEOUT, 5000).

Of course, that's not possible right now as you can't compile from source.

That helps

msmith — Wed, 09 Nov 2005 21:50:36 +0000

I can do the workaround by going to http://host:5280/admin/server/host/ and it pulls up shared roster and all is good. If I click on Users, it crashes again. I'm assuming its trying to pull up a list of User from LDAP? Maybe it can't handle all the users in my LDAP tree (~13,000)?

I compiled ejabberd (0.9.8) from source, but I couldn't get it to run. It just always produces a erl_crash.dump that is really big.

I've read something about this when doing a Google search -- it seems other(s) have had this problem in the past:

http://lists.jabber.ru/pipermail/ejabberd/2004-July/000154.html

Thanks,

Marc

Try this workaround

mfoss — Sun, 06 Nov 2005 21:50:54 +0000

Something about ldap appears there. It could be a bug related to LDAP, but in that case somebody else would have found it previously, and this is the first time I read about it.

You use the FreeBSD port, but I doubt that port introduced any new code that could produce the error. One thing you could try is to compile ejabberd from source.

Workaround: if you are lucky, the bug only affects '/admin/vhosts/'. If so, maybe you can access the virtual host directly at '/admin/server/chatty.mcc.edu/'.

ejabberd - Comments for "Can&#039;t Access Virtual Hosts Page"

LDAP w/ lots of users, solution

erlang CMD_TIMEOUT limit?

Won't Patch Correctly

some help for debugging

Does TIMEOUT variable have a ceiling?

Advanced LDAP config in ejabberd.cfg?

Check your LDAP server logs and settings

Works fine with ~100 users

Doesn't seem to help

Maybe it can't handle all

That helps

Try this workaround

ejabberd - Comments for "Can't Access Virtual Hosts Page"