Investigate the problem trying different ODBC pool sizes

mfoss — Tue, 12 Feb 2008 09:54:39 +0000

I can't help you in this problem, but maybe this gives you ideas to investigate the problem.

In ejabberd 1.1.4 and older, the number of SQL connections per host (which is 10 by default) is in the file ejabberd/src/odbc/ejabberd_odbc_sup.erl:

init([Host]) ->
    % TODO
    N = 10,

In ejabberd 2.0.0-rc1, this can be configured in ejabberd.cfg, so no need to recompile:

Quote:

By default ejabberd opens 10 connections to the database for each virtual host. Use this option to modify the value:

{odbc_pool_size, 10}.

You can configure an interval to make a dummy SQL request to keep alive the connections to the database. The default value is ’undefined’, so no keepalive requests are made. Specify in seconds: for example 28800 means 8 hours.

{odbc_keepalive_interval, undefined}.

Updated - file system limits

tlaramie — Mon, 10 Dec 2007 17:05:54 +0000

It turns out that the ulimit on the server may have been causing the problems though I luckily happen to catch it as it happened. Increasing the ulimit allowed all the MySQL processes to go through (540 total) but now there's a new problem:

(ejabberd@van-ejabberd1.globalrelay.net)1> [05:01 PM] van-ejabberd1:/opt/etc/ejabberd$ tail /tmp/start 
                               start,
                               [5223,
                                ejabberd_c2s,
                                [{access,c2s},
                                 tls,
                                 {certfile,"/opt/etc/ejabberd/server.pem"}]]}},
                       {restart_type,transient},
                       {shutdown,brutal_kill},
                       {child_type,worker}]
(ejabberd@van-ejabberd1.globalrelay.net)1> [05:01 PM] van-ejabberd1:/opt/etc/ejabberd$ tail -100 /tmp/start 
    registered_name: []
    error_info: {{badmatch,
                      {error,
                          {{badmatch,
                               {error,
                                   "SSL_CTX_use_certificate_file failed: error:02001018:system library:fopen:Too many open files"}},
                           [{ejabberd_c2s,init,1},
                            {gen_fsm,init_it,6},
                            {proc_lib,init_p,5}]}}},
                  [{ejabberd_listener,accept,3},{proc_lib,init_p,5}]}
    initial_call: {ejabberd_listener,
                     init,
                     [5223,
                      ejabberd_c2s,
                      [{access,c2s},
                       tls,
                       {certfile,"/opt/etc/ejabberd/server.pem"}]]}
    ancestors: [ejabberd_listeners,ejabberd_sup,<0.38.0>]
    messages: []
    links: [#Port<0.993>,<0.212.0>,#Port<0.287>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 377
    stack_size: 21
    reductions: 603
  neighbours:
(ejabberd@van-ejabberd1.globalrelay.net)1> 
=CRASH REPORT==== 10-Dec-2007::17:01:38 ===
  crasher:
    pid: <0.4552.0>
    registered_name: []
    error_info: {{badmatch,{error,"SSL_CTX_use_certificate_file failed: error:02001018:system library:fopen:Too many open files"}},
                  [{ejabberd_c2s,init,1},
                   {gen_fsm,init_it,6},
                   {proc_lib,init_p,5}]}
    initial_call: {gen,init_it,
                      [gen_fsm,
                       <0.214.0>,
                       self,
                       ejabberd_c2s,
                       [{gen_tcp,#Port<0.993>},
                        [{access,c2s},
                         tls,
                         {certfile,"/opt/etc/ejabberd/server.pem"}]],
                       []]}
    ancestors: [<0.214.0>,ejabberd_listeners,ejabberd_sup,<0.38.0>]
    messages: []
    links: [#Port<0.997>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 21
    reductions: 2082
  neighbours:
(ejabberd@van-ejabberd1.globalrelay.net)1> 
=SUPERVISOR REPORT==== 10-Dec-2007::17:01:38 ===
     Supervisor: {local,ejabberd_listeners}
     Context:    child_terminated
     Reason:     {{badmatch,
                      {error,
                          {{badmatch,
                               {error,
                                   "SSL_CTX_use_certificate_file failed: error:02001018:system library:fopen:Too many open files"}},
                           [{ejabberd_c2s,init,1},
                            {gen_fsm,init_it,6},
                            {proc_lib,init_p,5}]}}},
                  [{ejabberd_listener,accept,3},{proc_lib,init_p,5}]}
     Offender:   [{pid,<0.214.0>},
                  {name,5223},
                  {mfa,
                      {ejabberd_listener,
                          start,
                          [5223,
                           ejabberd_c2s,
                           [{access,c2s},
                            tls,
                            {certfile,"/opt/etc/ejabberd/server.pem"}]]}},
                  {restart_type,transient},
                  {shutdown,brutal_kill},
                  {child_type,worker}]

(ejabberd@van-ejabberd1.globalrelay.net)1> 
=PROGRESS REPORT==== 10-Dec-2007::17:01:38 ===
          supervisor: {local,ejabberd_listeners}
             started: [{pid,<0.4553.0>},
                       {name,5223},
                       {mfa,
                           {ejabberd_listener,
                               start,
                               [5223,
                                ejabberd_c2s,
                                [{access,c2s},
                                 tls,
                                 {certfile,"/opt/etc/ejabberd/server.pem"}]]}},
                       {restart_type,transient},
                       {shutdown,brutal_kill},
                       {child_type,worker}]

I've googled this error and variants and see some similar situations but nothing resolved (at least least in situations where ulimit has already been increased significantly.

$ ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 65535
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 10240
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited
lsof for ejabberd's user shows 1790+ though this is still well below what the system is set to allow.

ejabberd - Comments for "MySQL Connection Pool Exhaustion"

Investigate the problem trying different ODBC pool sizes

Updated - file system limits