ejabberd - Comments for "clustering questions"

/etc/hosts should work

mfoss — Thu, 30 Sep 2010 11:36:39 +0000

calestyo wrote:

Guess it's simply the IP that resolves from the erlang node name (-name or sname parameter), right?
So If I overwrite those domainnames in /etc/hosts, I should be able to use the internal IPs for communication...?

Yes. I usually add fake erlang node names and jabber domain names to my /etc/hosts, and both jabber clients and erlang clustering use that information correctly.

Time to start testing and stop asking

mfoss — Thu, 30 Sep 2010 11:33:25 +0000

I recommend you to wake up, pick several papers, a red pen for questions, a blue pen for answers and a fluorescent for marking the conclusions. As you read this thread, for every unanswered/dubitous question/problem, write the question in red in a new paper. Then go and install ejabberd, setup the cluster, test, set the additional ipsec, test, set the certs, test, try s2s to other jabber servers, text, ... Fill the papers with answers, partial answers and additional details. After a few days you will become an expert in your kind of cluster setup.

calestyo wrote:

The certs are _always_ for the served domain, never for the hostname, where the server actually points or where the SRV RR points to?

I don't know.

calestyo wrote:

Is it ok, if I simply select everything in "{tables,[c...}"? Or are some of them _just_ data for one local node?
Is there some explanation of what each table "does"?

Don't replicate all the tables. Not all tables have information that you want to be shared across several nodes.

See tutorials about cluser in
http://www.ejabberd.im/tutorials

calestyo wrote:

And tables that are not replicated,... if e.g. table xyz is on server a but not replicated to server b.
Now if a client requests data from xyz via server b,... it is not replicated, but just transferred, right?

No, because "not replicated" means "not replicated at all".

calestyo wrote:

Regarding "having the very same config file" on each node, you mentioned the MUC service. Isn't it possible to have all those services (e.g. MUC) twice on my two cluster nodes, serving the very same domains.
E.g. a MUC service on server A and B both serving conference.example.org

Clustering in ejabberd works by means of mnesia clustering, that is table sharing. In older ejabberd releases, MUC service didn't support clustering (only one instance could be running in the nodes of the cluster).

But I've right now noticed that this restriction was removed for newer ejabberd releases this year. So, you can try to enable mod_muc similarly in all the nodes.

calestyo wrote:

That will solve the security problem.

About connection reliability: you want to test what happens when the connection between the nodes gets lost.

About connection delay/lag: measure the delay seen by users when changing data in a replicated table (primarily the roster).

calestyo wrote:

1) Is communication between clusters _ONLY_ happening via TCP 4369 or are there other ports or even UDP involved? If so, which ones, or where can I find/configure this information.

Check http://www.process-one.net/en/ejabberd/guide_en#firewall

Summary: 4369 is only for EPMD (like a DNS for erlang connections). In general the traffic goes over other random port numbers, but you can delimit the range in ejabberdctl.cfg.

calestyo wrote:

2) You also mentioned that latency is critical (especially for the DB synchronisation I guess),... do you think it would still work, if I e.g. server just a handfull (~10) users?

Yes, with only 10 concurrent users, I think ejabberd and mnesia will be completely bored. After all, access to replicated tables is only necessary at login (tables: passwd, roster, offline_msg), stanza routing (privacy, maybe offline_msg), ...

even one more ;)

calestyo — Wed, 29 Sep 2010 14:29:14 +0000

With VPN, I'll probably have to use some private IP address.

How are the hosts (names and IPs) determined, which are used by erlang for communication?

further question ;)

calestyo — Wed, 29 Sep 2010 12:00:58 +0000

badlop wrote:

The client has to receive a valid certificate for that domain and that machine.

From a theoretical point of view that was clear to me,... a cert is valid,... therefore it should work, whether it has the same fingerpint and key or not.

But it might well be that case, that many clients or even others servers cannot cope with this.
E.g. they cache a cert for a given vhostname, and if another wan (same DN, but different key data) is presented, they fail because the think something bad is going on.
Are there any such known clients/servers (I know e.g. that pidgin does some local caching of server certs).

But in the end, and this is more an additional question:
The certs are _always_ for the served domain, never for the hostname, where the server actually points or where the SRV RR points to?

badlop wrote:

You manually configure in the Mnesia database of each erlang node what tables you want to be automatically replicated.

Is it ok, if I simply select everything in "{tables,[c...}"? Or are some of them _just_ data for one local node?
Is there some explanation of what each table "does"?

You further mentioned that clustered servers should be in the same localnet.
Unfortunately this is not possible for me (cost reasons).
I planned to use strongswan / IPsec to secure that communication:
1) Is communication between clusters _ONLY_ happening via TCP 4369 or are there other ports or even UDP involved? If so, which ones, or where can I find/configure this information.
2) You also mentioned that latency is critical (especially for the DB synchronisation I guess),... do you think it would still work, if I e.g. server just a handfull (~10) users?

thx.

calestyo wrote: 1) What I

mfoss — Wed, 29 Sep 2010 09:13:37 +0000

calestyo wrote:

1) What I want in principle, is that two) or more servers, serve the same jabber domains. And if any of that servers fail client and server connections should use the other one.
At best this would happen automatically without interruption.
Wheter load balancing is also done (e.g. server/client connections use both servers if they're up) doesn't matter.

The TCP connections established with the closing machine will get closed. When the clients and remote servers try to connect again to "example.org", you must be sure your DNS server points them to a working machine that serves that domain.

calestyo wrote:

Do also remote server notice if one of my servers go down, and use another server?

Is this possible at all?

They notice because the TCP connection gets closed.

calestyo wrote:

I've read the guide about clustering and also some forum posts, but it's still now fully clear to me:
2) If I follow the guide, do the two mnesia database fully (and constantly) replicate them selves?

For example, after you configure in Mnesia to replicate the tables "roster" and "offline_msg" across 3 nodes, Mnesia itself takes care to have exactly the same content in those tables in the three nodes, and replicate the changes between the involved nodes. This way the ejabberds running in each node will see exactly the same rosters and the same offline messages.

calestyo wrote:

3) Do I have to use the same ejabbert.cfg on both nodes? What happens if I don't. E.g. different served hosts or different admin users on both nodes. Would they synchronise each other?

Obviously you want and can have slightly different cfg files in each node: in a node you enable mod_muc for the "example.org" domain, not in the others. In a node you can have additional acl rules, or different shapers for ejabberd_c2s listener.

Each ejabberd node is independent. The nodes only communicate between them by means of the Mnesia tables that you configured to be shared.

calestyo wrote:

4) How to set up certificates?
Right now I have set a default (???) certificate in the listeners with:
{certfile, "/etc/ejabberd/certificates/xmpp.srv.scientia.net"}
and per vhost certificates like:
{domain_certfile, "xmpp.srv.scientia.net", "/etc/ejabberd/certificates/xmpp.srv.scientia.net"}.
{domain_certfile, "scientia.net", "/etc/ejabberd/certificates/scientia.net"}.
{domain_certfile, "informatica.scientia.net", "/etc/ejabberd/certificates/informatica.scientia.net"}.
{domain_certfile, "christoph.anton.mitterer.name", "/etc/ejabberd/certificates/christoph.anton.mitterer.name"}.

The hosts will be named a.xmpp.srv.scientia.net and b.xmpp.srv.scientia.net however.
Do I need additional certs for the canonical hostname?

Or do both hosts need to use the same certificates?

I don't know. I imagine you have a certificate for the host "example.org", and you copy that same file to all the machines.

calestyo wrote:

4) The erlang node name (which is also in the DB) and the Jabber host names, are completely different, right? So I could use other erlang node names, than vhosts served, e.g. as node names the a.xmpp.srv.scientia.net and b.xmpp.srv.scientia.net?

Yes, erlang node name and Jabber domain name are independent topics. In each case put the correct/valid/desired values for each topic.

calestyo wrote:

5) How to configure DNS?
Simply make two SRV records, with the same priorities if I want load balancing or different priorities, if I want my "secondary" server to be just used in case the primary fails?

Yes, that's the general idea.

calestyo wrote:

6) The communication between the two cluster nodes, is it encrypted signed? (with the erlang cookie?)

The erlang cookie is a management tool, used to ensure that no wrong clustering is made in a network with many clusters with many nodes each; it isn't designed as a security measure. Also, the communication between erlang nodes isn't encrypted. And that communication must have low ping, and preferably not closed.

As you can see, the machines that have a cluster of erlang nodes should be in the same LAN, not in the open Internet.

Valid for that instance

mfoss — Wed, 29 Sep 2010 08:55:04 +0000

calestyo wrote:

The certs on the two nodes, for the different vhosts,... (I guess those are not replicated),... do the have to be exactly the same (same key) or just a valid cert with the same DN?

For TCP and XMPP matters, when a XMPP client connects to an ejabberd node, the client doesn't care if there are other machines that also serve the same domain. The client has to receive a valid certificate for that domain and that machine.

Manual setup

mfoss — Wed, 29 Sep 2010 08:52:47 +0000

calestyo wrote:

And is it possible to tell the DB to simply replicate _ALL_ tables automatically?
Will this include then offline messages?

You manually configure in the Mnesia database of each erlang node what tables you want to be automatically replicated.

and even more more question

calestyo — Tue, 28 Sep 2010 15:56:28 +0000

The certs on the two nodes, for the different vhosts,... (I guess those are not replicated),... do the have to be exactly the same (same key) or just a valid cert with the same DN?

Ah... And is it possible to

calestyo — Tue, 28 Sep 2010 12:45:22 +0000

Ah...
And is it possible to tell the DB to simply replicate _ALL_ tables automatically?
Will this include then offline messages?