ejabberd 2.1.4 crashes during stress test

I'm stress testing ejabberd and it keeps crashing when the connection terminate from ejabberd. The crash dump states that application terminated because mnesia was killed. I've increased the ERL_MAX_ETS_TABLES to 60000.

What is the recommended configuration value to handle very large numbers of connection are terminating?

Logs, dump files, and crash log excerpt is below.

#.
#' ERL_MAX_ETS_TABLES: Maximum number of ETS and Mnesia tables
#
# The number of concurrent ETS and Mnesia tables is limited. When the limit is
# reached, errors will appear in the logs:
# ** Too many db tables **
# You can safely increase this limit when starting ejabberd. It impacts memory
# consumption but the difference will be quite small.
#
# Default: 1400
#
ERL_MAX_ETS_TABLES=60000

In the ejabberd log file I see the following log lines.

=ERROR REPORT==== 2010-09-20 20:22:30 ===
Mnesia('ejabberd@ejab2.sjc.tango.me'): ** WARNING ** Mnesia is overloaded: {dump_log,
write_threshold}

=ERROR REPORT==== 2010-09-20 20:22:34 ===
Mnesia('ejabberd@ejab2.sjc.tango.me'): ** WARNING ** Mnesia is overloaded: {dump_log,
write_threshold}

=ERROR REPORT==== 2010-09-20 20:22:35 ===
Mnesia('ejabberd@ejab2.sjc.tango.me'): ** WARNING ** Mnesia is overloaded: {dump_log,
write_threshold}

...

=ERROR REPORT==== 2010-09-20 21:00:22 ===
** Too many db tables **

=ERROR REPORT==== 2010-09-20 21:00:22 ===
** Too many db tables **

=ERROR REPORT==== 2010-09-20 21:00:22 ===
** Generic server mnesia_monitor terminating
** Last message in was {'EXIT',<0.60.0>,killed}
** When Server state == {state,<0.60.0>,[],[],true,[],undefined,[]}
** Reason for termination ==
** killed

=INFO REPORT==== 2010-09-20 21:00:23 ===
application: mnesia
exited: killed
type: permanent

=ERROR REPORT==== 2010-09-20 21:00:24 ===
application_master: shutdown_error
ejabberd_app: {prep_stop,[[]]}
error_info: {badarg,[{ets,lookup,[config,hosts]},
{ejabberd_config,get_global_option,1},
{ejabberd_app,stop_modules,0},
{ejabberd_app,prep_stop,1},
{application_master,prep_stop,2},
{application_master,loop_it,4}]}

-------------------- From the crash dump --------------------

=erl_crash_dump:0.1
Mon Sep 20 21:00:26 2010
Slogan: Kernel pid terminated (application_controller) ({application_terminated,mnesia,killed})
System version: Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:true]
Compiled: Wed Aug 25 16:05:49 2010
Taints:
Atoms: 14002
=memory
total: 915806376
processes: 244769320
processes_used: 57010856
system: 671037056
atom: 955065
atom_used: 944216
binary: 24888
code: 11349780
ets: 128389368
=hash_table:atom_tab
size: 9643
used: 7363
objs: 14002
depth: 7
=index_table:atom_tab
size: 14336
limit: 1048576
entries: 14002
=hash_table:module_code
size: 197
used: 148
objs: 241
depth: 4
=index_table:module_code
size: 1024
limit: 65536
entries: 241
=hash_table:export_list
size: 4813
used: 3336
objs: 5651

Re: ejabberd 2.1.4 crashes during stress test

1) It seems like even 60000 is not enough, try to increase.
2) You should use SQL on the heavily loaded server (to avoid those mnesia warnings).

We are using mysql through

We are using mysql through the odbc driver. I'm not sure what ejabberd writes to mnesia when using odbc drivers.

I've increased it to 200000 and I'm still seeing errors.

Stuck at 250K connections

You are seeing those errors

You are seeing those errors because a mnesia creates an ets table per transaction. BTW, I don't know if odbc driver uses mnesia or not, and I *highly* recommend to use mysql driver.
Also, could you please show your config?

we're using mysql odbc

we're using mysql odbc driver.

Here is the config file sans passwords

%%%
%%% ejabberd configuration file
%%%
%%%'

%%% The parameters used in this configuration file are explained in more detail
%%% in the ejabberd Installation and Operation Guide.
%%% Please consult the Guide in case of doubts, it is included with
%%% your copy of ejabberd, and is also available online at
%%% http://www.process-one.net/en/ejabberd/docs/

%%% This configuration file contains Erlang terms.
%%% In case you want to understand the syntax, here are the concepts:
%%%
%%% - The character to comment a line is %
%%%
%%% - Each term ends in a dot, for example:
%%% override_global.
%%%
%%% - A tuple has a fixed definition, its elements are
%%% enclosed in {}, and separated with commas:
%%% {loglevel, 4}.
%%%
%%% - A list can have as many elements as you want,
%%% and is enclosed in [], for example:
%%% [http_poll, web_admin, tls]
%%%
%%% - A keyword of ejabberd is a word in lowercase.
%%% Strings are enclosed in "" and can contain spaces, dots, ...
%%% {language, "en"}.
%%% {ldap_rootdn, "dc=example,dc=com"}.
%%%
%%% - This term includes a tuple, a keyword, a list, and two strings:
%%% {hosts, ["jabber.example.net", "im.example.com"]}.
%%%

%%%. =======================
%%%' OVERRIDE STORED OPTIONS

%%
%% Override the old values stored in the database.
%%

%%
%% Override global options (shared by all ejabberd nodes in a cluster).
%%
%%override_global.

%%
%% Override local options (specific for this particular ejabberd node).
%%
%%override_local.

%%
%% Remove the Access Control Lists before new ones are added.
%%
%%override_acls.

%%%. =========
%%%' DEBUGGING

%%
%% loglevel: Verbosity of log files generated by ejabberd.
%% 0: No ejabberd log at all (not recommended)
%% 1: Critical
%% 2: Error
%% 3: Warning
%% 4: Info
%% 5: Debug
%%
{loglevel, 5}.

%%
%% watchdog_admins: Only useful for developers: if an ejabberd process
%% consumes a lot of memory, send live notifications to these XMPP
%% accounts.
%%
%%{watchdog_admins, ["bob@example.com"]}.

%%%. ================
%%%' SERVED HOSTNAMES

%%
%% hosts: Domains served by ejabberd.
%% You can define one or several, for example:
%% {hosts, ["example.net", "example.com", "example.org"]}.
%%
{hosts, ["realtime.sgiggle.com"]}.

%%
%% route_subdomains: Delegate subdomains to other XMPP servers.
%% For example, if this ejabberd serves example.org and you want
%% to allow communication with an XMPP server called im.example.org.
%%
%%{route_subdomains, s2s}.

%%%. ===============
%%%' LISTENING PORTS

%%
%% listen: The ports ejabberd will listen on, which service each is handled
%% by and what options to start it with.
%%
{listen,
[

{6222, ejabberd_c2s, [

%%
%% If TLS is compiled in and you installed a SSL
%% certificate, specify the full path to the
%% file and uncomment this line:
%%
{certfile, "/local/ejabberd/etc/ejabberd/jabber.pem"}, starttls_required,

{access, c2s},
{shaper, c2s_shaper},
{max_stanza_size, 1048576}
]},

%%{5269, ejabberd_s2s_in, [
%% {shaper, s2s_shaper},
%% {max_stanza_size, 131072}
%% ]},

%%
%% ejabberd_service: Interact with external components (transports, ...)
%%
%%{8888, ejabberd_service, [
%% {access, all},
%% {shaper_rule, fast},
%% {ip, {127, 0, 0, 1}},
%% {hosts, ["icq.example.org", "sms.example.org"],
%% [{password, "secret"}]
%% }
%% ]},

%%
%% ejabberd_stun: Handles STUN Binding requests
%%
%%{{3478, udp}, ejabberd_stun, []},

{5280, ejabberd_http, [
%%{request_handlers,
%% [
%% {["pub", "archive"], mod_http_fileserver}
%% ]},
captcha,
http_bind,
http_poll,
web_admin
]}

]}.

%%
%% s2s_use_starttls: Enable STARTTLS + Dialback for S2S connections.
%% Allowed values are: true or false.
%% You must specify a certificate file.
%%
%%{s2s_use_starttls, true}.

%%
%% s2s_certfile: Specify a certificate file.
%%
%%{s2s_certfile, "/path/to/ssl.pem"}.

%%
%% domain_certfile: Specify a different certificate for each served hostname.
%%
%%{domain_certfile, "example.org", "/path/to/example_org.pem"}.
%%{domain_certfile, "example.com", "/path/to/example_com.pem"}.

%%
%% S2S whitelist or blacklist
%%
%% Default s2s policy for undefined hosts.
%%
%%{s2s_default_policy, allow}.

%%
%% Allow or deny communication with specific servers.
%%
%%{{s2s_host, "goodhost.org"}, allow}.
%%{{s2s_host, "badhost.org"}, deny}.

%%
%% Outgoing S2S options
%%
%% Preferred address families (which to try first) and connect timeout
%% in milliseconds.
%%
%%{outgoing_s2s_options, [ipv4, ipv6], 10000}.

%%%. ==============
%%%' AUTHENTICATION

%%
%% auth_method: Method used to authenticate the users.
%% The default method is the internal.
%% If you want to use a different method,
%% comment this line and enable the correct ones.
%%
%%{auth_method, internal}.

%%
%% Authentication using external script
%% Make sure the script is executable by ejabberd.
%%
%%{auth_method, external}.
%%{extauth_program, "/path/to/authentication/script"}.

%%
%% Authentication using ODBC
%% Remember to setup a database in the next section.
%%
{auth_method, odbc}.

%%
%% Authentication using PAM
%%
%%{auth_method, pam}.
%%{pam_service, "pamservicename"}.

%%
%% Authentication using LDAP
%%
%%{auth_method, ldap}.
%%
%% List of LDAP servers:
%%{ldap_servers, ["localhost"]}.
%%
%% Encryption of connection to LDAP servers:
%%{ldap_encrypt, none}.
%%{ldap_encrypt, tls}.
%%
%% Port to connect to on LDAP servers:
%%{ldap_port, 389}.
%%{ldap_port, 636}.
%%
%% LDAP manager:
%%{ldap_rootdn, "dc=example,dc=com"}.
%%
%% Password of LDAP manager:
%%{ldap_password, "******"}.
%%
%% Search base of LDAP directory:
%%{ldap_base, "dc=example,dc=com"}.
%%
%% LDAP attribute that holds user ID:
%%{ldap_uids, [{"mail", "%u@mail.example.org"}]}.
%%
%% LDAP filter:
%%{ldap_filter, "(objectClass=shadowAccount)"}.

%%
%% Anonymous login support:
%% auth_method: anonymous
%% anonymous_protocol: sasl_anon | login_anon | both
%% allow_multiple_connections: true | false
%%
%%{host_config, "public.example.org", [{auth_method, anonymous},
%% {allow_multiple_connections, false},
%% {anonymous_protocol, sasl_anon}]}.
%%
%% To use both anonymous and internal authentication:
%%
%%{host_config, "public.example.org", [{auth_method, [internal, anonymous]}]}.
%%{host_config, "realtime.sgiggle.com", [{auth_method, [internal,anonymous]}, {anonymous_protocol, sasl_anon}]}.

%%%. ==============
%%%' DATABASE SETUP

%% ejabberd by default uses the internal Mnesia database,
%% so you do not necessarily need this section.
%% This section provides configuration examples in case
%% you want to use other database backends.
%% Please consult the ejabberd Guide for details on database creation.

%%
%% MySQL server:
%%
{odbc_server, {mysql, server, 3306, db, user, password}}.
%%
%% If you want to specify the port:
%%{odbc_server, {mysql, "server", 1234, "database", "username", "password"}}.

%%
%% PostgreSQL server:
%%
%%{odbc_server, {pgsql, "server", "database", "username", "password"}}.
%%
%% If you want to specify the port:
%%{odbc_server, {pgsql, "server", 1234, "database", "username", "password"}}.
%%
%% If you use PostgreSQL, have a large database, and need a
%% faster but inexact replacement for "select count(*) from users"
%%
%%{pgsql_users_number_estimate, true}.

%%
%% ODBC compatible or MSSQL server:
%%
%%{odbc_server, "DSN=ejabberd;UID=ejabberd;PWD=ejabberd"}.

%%
%% Number of connections to open to the database for each virtual host
%%
%%{odbc_pool_size, 10}.

%%
%% Interval to make a dummy SQL request to keep the connections to the
%% database alive. Specify in seconds: for example 28800 means 8 hours
%%
%%{odbc_keepalive_interval, undefined}.

%%%. ===============
%%%' TRAFFIC SHAPERS

%%
%% The "normal" shaper limits traffic speed to 1000 B/s
%%
{shaper, normal, {maxrate, 50000000}}.

%%
%% The "fast" shaper limits traffic speed to 50000 B/s
%%
{shaper, fast, {maxrate, 50000000}}.

%%
%% This option specifies the maximum number of elements in the queue
%% of the FSM. Refer to the documentation for details.
%%
{max_fsm_queue, 10000}.

%%%. ====================
%%%' ACCESS CONTROL LISTS

%%
%% The 'admin' ACL grants administrative privileges to XMPP accounts.
%% You can put here as many accounts as you want.
%%
%%{acl, admin, {user, "aleksey", "localhost"}}.
%%{acl, admin, {user, "ermine", "example.org"}}.

%%
%% Blocked users
%%
%%{acl, blocked, {user, "baduser", "example.org"}}.
%%{acl, blocked, {user, "test"}}.

%%
%% Local users: don't modify this line.
%%
{acl, local, {user_regexp, ""}}.

%%
%% More examples of ACLs
%%
%%{acl, jabberorg, {server, "jabber.org"}}.
%%{acl, aleksey, {user, "aleksey", "jabber.ru"}}.
%%{acl, test, {user_regexp, "^test"}}.
%%{acl, test, {user_glob, "test*"}}.

%%
%% Define specific ACLs in a virtual host.
%%
{host_config, "realtime.sgiggle.com",
[
{acl, admin, {user, "admin", "realtime.sgiggle.com"}}
]
}.

%%%. ============
%%%' ACCESS RULES

%% Maximum number of simultaneous sessions allowed for a single user:
{access, max_user_sessions, [{10, all}]}.

%% Maximum number of offline messages that users can have:
{access, max_user_offline_messages, [{5000, admin}, {100, all}]}.

%% This rule allows access only for local users:
{access, local, [{allow, local}]}.

%% Only non-blocked users can use c2s connections:
{access, c2s, [{deny, blocked},
{allow, all}]}.

%% For C2S connections, all users except admins use the "normal" shaper
{access, c2s_shaper, [{none, admin},
{normal, all}]}.

%% All S2S connections use the "fast" shaper
{access, s2s_shaper, [{fast, all}]}.

%% Only admins can send announcement messages:
{access, announce, [{allow, admin}]}.

%% Only admins can use the configuration interface:
{access, configure, [{allow, admin}]}.

%% Admins of this server are also admins of the MUC service:
{access, muc_admin, [{allow, admin}]}.

%% Only accounts of the local ejabberd server can create rooms:
{access, muc_create, [{allow, local}]}.

%% All users are allowed to use the MUC service:
{access, muc, [{allow, all}]}.

%% Only accounts on the local ejabberd server can create Pubsub nodes:
{access, pubsub_createnode, [{allow, local}]}.

%% In-band registration allows registration of any possible username.
%% To disable in-band registration, replace 'allow' with 'deny'.
{access, register, [{allow, all}]}.

%% By default the frequency of account registrations from the same IP
%% is limited to 1 account every 10 minutes. To disable, specify: infinity
%%{registration_timeout, 600}.

%%
%% Define specific Access Rules in a virtual host.
%%
%%{host_config, "localhost",
%% [
%% {access, c2s, [{allow, admin}, {deny, all}]},
%% {access, register, [{deny, all}]}
%% ]
%%}.

%%%. ================
%%%' DEFAULT LANGUAGE

%%
%% language: Default language used for server messages.
%%
{language, "en"}.

%%
%% Set a different default language in a virtual host.
%%
%%{host_config, "localhost",
%% [{language, "ru"}]
%%}.

%%%. =======
%%%' CAPTCHA

%%
%% Full path to a script that generates the image.
%%
%%{captcha_cmd, "/lib/ejabberd/priv/bin/captcha.sh"}.

%%
%% Host part of the URL sent to the user.
%%
%%{captcha_host, "example.org:5280"}.

%%%. =======
%%%' MODULES

%%
%% Modules enabled in all ejabberd virtual hosts.
%%
{modules,
[
{mod_adhoc, []},
{mod_announce, [{access, announce}]}, % recommends mod_adhoc
{mod_caps, []},
{mod_configure,[]}, % requires mod_adhoc
{mod_disco, []},
%%{mod_echo, [{host, "echo.localhost"}]},
{mod_irc, []},
{mod_http_bind, []},
%%{mod_http_fileserver, [
%% {docroot, "/var/www"},
%% {accesslog, "/var/log/ejabberd/access.log"}
%% ]},
{mod_last, []},
{mod_muc, [
%%{host, "conference.@HOST@"},
{access, muc},
{access_create, muc_create},
{access_persistent, muc_create},
{access_admin, muc_admin}
]},
%%{mod_muc_log,[]},
{mod_offline, [{access_max_user_messages, max_user_offline_messages}]},
{mod_ping, [{send_pings, true}, {ping_interval, 900}, {timeout_action, kill}]},
{mod_privacy, []},
{mod_private, []},
%%{mod_proxy65,[]},
{mod_pubsub, [
{access_createnode, pubsub_createnode},
{ignore_pep_from_offline, true}, % reduces resource comsumption, but XEP incompliant
%%{ignore_pep_from_offline, false}, % XEP compliant, but increases resource comsumption
{last_item_cache, false},
{plugins, ["flat", "hometree", "pep"]} % pep requires mod_caps
]},
{mod_register, [
%%
%% After successful registration, the user receives
%% a message with this subject and body.
%%
{welcome_message, {"Welcome!",
"Hi.\nWelcome to this XMPP server."}},

%%
%% When a user registers, send a notification to
%% these XMPP accounts.
%%
%%{registration_watchers, ["admin1@example.org"]},

{access, register}
]},
{mod_roster, []},
%%{mod_service_log,[]},
{mod_shared_roster,[]},
{mod_stats, []},
{mod_time, []},
{mod_vcard, []},
{mod_version, []}
]}.

%%
%% Enable modules with custom options in a specific virtual host
%%
%%{host_config, "localhost",
%% [{{add, modules},
%% [
%% {mod_echo, [{host, "mirror.localhost"}]}
%% ]
%% }
%% ]}.

{registration_timeout, infinity}.

%%%.
%%%'

%%% $Id$

%%% Local Variables:
%%% mode: erlang
%%% End:
%%% vim: set filetype=erlang tabstop=8 foldmarker=%%%',%%%. foldmethod=marker:

ejabberdctl.cfg

#
# In this file you can configure options that are passed by ejabberdctl
# to the erlang runtime system when starting ejabberd
#

#' POLL: Kernel polling ([true|false])
#
# The kernel polling option requires support in the kernel.
# Additionally, you need to enable this feature while compiling Erlang.
#
# Default: true
#
POLL=true

#.
#' SMP: SMP support ([enable|auto|disable])
#
# Explanation in Erlang/OTP documentation:
# enable: starts the Erlang runtime system with SMP support enabled.
# This may fail if no runtime system with SMP support is available.
# auto: starts the Erlang runtime system with SMP support enabled if it
# is available and more than one logical processor are detected.
# disable: starts a runtime system without SMP support.
#
# Default: auto
#
SMP=enable

#.
#' ERL_MAX_PORTS: Maximum number of simultaneously open Erlang ports
#
# ejabberd consumes two or three ports for every connection, either
# from a client or from another Jabber server. So take this into
# account when setting this limit.
#
# Default: 32000
# Maximum: 268435456
#
ERL_MAX_PORTS=5000000

#.
#' FIREWALL_WINDOW: Range of allowed ports to pass through a firewall
#
# If Ejabberd is configured to run in cluster, and a firewall is blocking ports,
# it's possible to make Erlang use a defined range of port (instead of dynamic
# ports) for node communication.
#
# Default: not defined
# Example: 4200-4210
#
#FIREWALL_WINDOW=9000-9100

#.
#' ERL_PROCESSES: Maximum number of Erlang processes
#
# Erlang consumes a lot of lightweight processes. If there is a lot of activity
# on ejabberd so that the maximum number of processes is reached, people will
# experiment greater latency times. As these processes are implemented in
# Erlang, and therefore not related to the operating system processes, you do
# not have to worry about allowing a huge number of them.
#
# Default: 250000
# Maximum: 268435456
#
ERL_PROCESSES=5000000

#.
#' ERL_MAX_ETS_TABLES: Maximum number of ETS and Mnesia tables
#
# The number of concurrent ETS and Mnesia tables is limited. When the limit is
# reached, errors will appear in the logs:
# ** Too many db tables **
# You can safely increase this limit when starting ejabberd. It impacts memory
# consumption but the difference will be quite small.
#
# Default: 1400
#
ERL_MAX_ETS_TABLES=40000

#.
#' ERL_OPTIONS: Additional Erlang options
#
# The next variable allows to specify additional options passed to erlang while
# starting ejabberd. Some useful options are -noshell, -detached, -heart. When
# ejabberd is started from an init.d script options -noshell and -detached are
# added implicitly. See erl(1) for more info.
#
# It might be useful to add "-pa /usr/local/lib/ejabberd/ebin" if you
# want to add local modules in this path.
#
# Default: ""
#
#ERL_OPTIONS=""

#.
#' ERLANG_NODE: Erlang node name
#
# The next variable allows to explicitly specify erlang node for ejabberd
# It can be given in different formats:
# ERLANG_NODE=ejabberd
# Lets erlang add hostname to the node (ejabberd uses short name in this case)
# ERLANG_NODE=ejabberd@hostname
# Erlang uses node name as is (so make sure that hostname is a real
# machine hostname or you'll not be able to control ejabberd)
# ERLANG_NODE=ejabberd@hostname.domainname
# The same as previous, but erlang will use long hostname
# (see erl (1) manual for details)
#
# Default: ejabberd
#
ERLANG_NODE=

#.
#' EJABBERD_PID_PATH: ejabberd PID file
#
# Indicate the full path to the ejabberd Process identifier (PID) file.
# If this variable is defined, ejabberd writes the PID file when starts,
# and deletes it when stops.
# Remember to create the directory and grant write permission to ejabberd.
#
# Default: don't write PID file
#
#EJABBERD_PID_PATH=/var/run/ejabberd/ejabberd.pid

#.
#'
# vim: foldmarker=#',#. foldmethod=marker:

Syndicate content