Hi.
I'd have several questions about ejabberd's clustering:
1) What I want in principle, is that two) or more servers, serve the same jabber domains. And if any of that servers fail client and server connections should use the other one.
At best this would happen automatically without interruption.
Wheter load balancing is also done (e.g. server/client connections use both servers if they're up) doesn't matter.
Do also remote server notice if one of my servers go down, and use another server?
Is this possible at all?
I've read the guide about clustering and also some forum posts, but it's still now fully clear to me:
2) If I follow the guide, do the two mnesia database fully (and constantly) replicate them selves?
3) Do I have to use the same ejabbert.cfg on both nodes? What happens if I don't. E.g. different served hosts or different admin users on both nodes. Would they synchronise each other?
4) How to set up certificates?
Right now I have set a default (???) certificate in the listeners with:
{certfile, "/etc/ejabberd/certificates/xmpp.srv.scientia.net"}
and per vhost certificates like:
{domain_certfile, "xmpp.srv.scientia.net", "/etc/ejabberd/certificates/xmpp.srv.scientia.net"}.
{domain_certfile, "scientia.net", "/etc/ejabberd/certificates/scientia.net"}.
{domain_certfile, "informatica.scientia.net", "/etc/ejabberd/certificates/informatica.scientia.net"}.
{domain_certfile, "christoph.anton.mitterer.name", "/etc/ejabberd/certificates/christoph.anton.mitterer.name"}.
The hosts will be named a.xmpp.srv.scientia.net and b.xmpp.srv.scientia.net however.
Do I need additional certs for the canonical hostname?
Or do both hosts need to use the same certificates?
4) The erlang node name (which is also in the DB) and the Jabber host names, are completely different, right? So I could use other erlang node names, than vhosts served, e.g. as node names the a.xmpp.srv.scientia.net and b.xmpp.srv.scientia.net?
5) How to configure DNS?
Simply make two SRV records, with the same priorities if I want load balancing or different priorities, if I want my "secondary" server to be just used in case the primary fails?
6) The communication between the two cluster nodes, is it encrypted signed? (with the erlang cookie?)
7) Anything else to consider?
Thanks,
Chris.
Ah... And is it possible to
Ah...
And is it possible to tell the DB to simply replicate _ALL_ tables automatically?
Will this include then offline messages?
and even more more question
The certs on the two nodes, for the different vhosts,... (I guess those are not replicated),... do the have to be exactly the same (same key) or just a valid cert with the same DN?
Valid for that instance
The certs on the two nodes, for the different vhosts,... (I guess those are not replicated),... do the have to be exactly the same (same key) or just a valid cert with the same DN?
For TCP and XMPP matters, when a XMPP client connects to an ejabberd node, the client doesn't care if there are other machines that also serve the same domain. The client has to receive a valid certificate for that domain and that machine.
further question ;)
The client has to receive a valid certificate for that domain and that machine.
From a theoretical point of view that was clear to me,... a cert is valid,... therefore it should work, whether it has the same fingerpint and key or not.
But it might well be that case, that many clients or even others servers cannot cope with this.
E.g. they cache a cert for a given vhostname, and if another wan (same DN, but different key data) is presented, they fail because the think something bad is going on.
Are there any such known clients/servers (I know e.g. that pidgin does some local caching of server certs).
But in the end, and this is more an additional question:
The certs are _always_ for the served domain, never for the hostname, where the server actually points or where the SRV RR points to?
You manually configure in the Mnesia database of each erlang node what tables you want to be automatically replicated.
Is it ok, if I simply select everything in "{tables,[c...}"? Or are some of them _just_ data for one local node?
Is there some explanation of what each table "does"?
And tables that are not replicated,... if e.g. table xyz is on server a but not replicated to server b.
Now if a client requests data from xyz via server b,... it is not replicated, but just transferred, right?
Regarding "having the very same config file" on each node, you mentioned the MUC service. Isn't it possible to have all those services (e.g. MUC) twice on my two cluster nodes, serving the very same domains.
E.g. a MUC service on server A and B both serving conference.example.org
You further mentioned that clustered servers should be in the same localnet.
Unfortunately this is not possible for me (cost reasons).
I planned to use strongswan / IPsec to secure that communication:
1) Is communication between clusters _ONLY_ happening via TCP 4369 or are there other ports or even UDP involved? If so, which ones, or where can I find/configure this information.
2) You also mentioned that latency is critical (especially for the DB synchronisation I guess),... do you think it would still work, if I e.g. server just a handfull (~10) users?
thx.
even one more ;)
With VPN, I'll probably have to use some private IP address.
How are the hosts (names and IPs) determined, which are used by erlang for communication?
Guess it's simply the IP that resolves from the erlang node name (-name or sname parameter), right?
So If I overwrite those domainnames in /etc/hosts, I should be able to use the internal IPs for communication...?
/etc/hosts should work
Guess it's simply the IP that resolves from the erlang node name (-name or sname parameter), right?
So If I overwrite those domainnames in /etc/hosts, I should be able to use the internal IPs for communication...?
Yes. I usually add fake erlang node names and jabber domain names to my /etc/hosts, and both jabber clients and erlang clustering use that information correctly.
Time to start testing and stop asking
I recommend you to wake up, pick several papers, a red pen for questions, a blue pen for answers and a fluorescent for marking the conclusions. As you read this thread, for every unanswered/dubitous question/problem, write the question in red in a new paper. Then go and install ejabberd, setup the cluster, test, set the additional ipsec, test, set the certs, test, try s2s to other jabber servers, text, ... Fill the papers with answers, partial answers and additional details. After a few days you will become an expert in your kind of cluster setup.
The certs are _always_ for the served domain, never for the hostname, where the server actually points or where the SRV RR points to?
I don't know.
Is it ok, if I simply select everything in "{tables,[c...}"? Or are some of them _just_ data for one local node?
Is there some explanation of what each table "does"?
Don't replicate all the tables. Not all tables have information that you want to be shared across several nodes.
See tutorials about cluser in
http://www.ejabberd.im/tutorials
And tables that are not replicated,... if e.g. table xyz is on server a but not replicated to server b.
Now if a client requests data from xyz via server b,... it is not replicated, but just transferred, right?
No, because "not replicated" means "not replicated at all".
Regarding "having the very same config file" on each node, you mentioned the MUC service. Isn't it possible to have all those services (e.g. MUC) twice on my two cluster nodes, serving the very same domains.
E.g. a MUC service on server A and B both serving conference.example.org
Clustering in ejabberd works by means of mnesia clustering, that is table sharing. In older ejabberd releases, MUC service didn't support clustering (only one instance could be running in the nodes of the cluster).
But I've right now noticed that this restriction was removed for newer ejabberd releases this year. So, you can try to enable mod_muc similarly in all the nodes.
You further mentioned that clustered servers should be in the same localnet.
Unfortunately this is not possible for me (cost reasons).
I planned to use strongswan / IPsec to secure that communication:
That will solve the security problem.
About connection reliability: you want to test what happens when the connection between the nodes gets lost.
About connection delay/lag: measure the delay seen by users when changing data in a replicated table (primarily the roster).
1) Is communication between clusters _ONLY_ happening via TCP 4369 or are there other ports or even UDP involved? If so, which ones, or where can I find/configure this information.
Check
Summary: 4369 is only for EPMD (like a DNS for erlang connections). In general the traffic goes over other random port numbers, but you can delimit the range in ejabberdctl.cfg.
2) You also mentioned that latency is critical (especially for the DB synchronisation I guess),... do you think it would still work, if I e.g. server just a handfull (~10) users?
Yes, with only 10 concurrent users, I think ejabberd and mnesia will be completely bored. After all, access to replicated tables is only necessary at login (tables: passwd, roster, offline_msg), stanza routing (privacy, maybe offline_msg), ...
Manual setup
And is it possible to tell the DB to simply replicate _ALL_ tables automatically?
Will this include then offline messages?
You manually configure in the Mnesia database of each erlang node what tables you want to be automatically replicated.
calestyo wrote: 1) What I
1) What I want in principle, is that two) or more servers, serve the same jabber domains. And if any of that servers fail client and server connections should use the other one.
At best this would happen automatically without interruption.
Wheter load balancing is also done (e.g. server/client connections use both servers if they're up) doesn't matter.
The TCP connections established with the closing machine will get closed. When the clients and remote servers try to connect again to "example.org", you must be sure your DNS server points them to a working machine that serves that domain.
Do also remote server notice if one of my servers go down, and use another server?
Is this possible at all?
They notice because the TCP connection gets closed.
I've read the guide about clustering and also some forum posts, but it's still now fully clear to me:
2) If I follow the guide, do the two mnesia database fully (and constantly) replicate them selves?
For example, after you configure in Mnesia to replicate the tables "roster" and "offline_msg" across 3 nodes, Mnesia itself takes care to have exactly the same content in those tables in the three nodes, and replicate the changes between the involved nodes. This way the ejabberds running in each node will see exactly the same rosters and the same offline messages.
3) Do I have to use the same ejabbert.cfg on both nodes? What happens if I don't. E.g. different served hosts or different admin users on both nodes. Would they synchronise each other?
Obviously you want and can have slightly different cfg files in each node: in a node you enable mod_muc for the "example.org" domain, not in the others. In a node you can have additional acl rules, or different shapers for ejabberd_c2s listener.
Each ejabberd node is independent. The nodes only communicate between them by means of the Mnesia tables that you configured to be shared.
4) How to set up certificates?
Right now I have set a default (???) certificate in the listeners with:
{certfile, "/etc/ejabberd/certificates/xmpp.srv.scientia.net"}
and per vhost certificates like:
{domain_certfile, "xmpp.srv.scientia.net", "/etc/ejabberd/certificates/xmpp.srv.scientia.net"}.
{domain_certfile, "scientia.net", "/etc/ejabberd/certificates/scientia.net"}.
{domain_certfile, "informatica.scientia.net", "/etc/ejabberd/certificates/informatica.scientia.net"}.
{domain_certfile, "christoph.anton.mitterer.name", "/etc/ejabberd/certificates/christoph.anton.mitterer.name"}.
The hosts will be named a.xmpp.srv.scientia.net and b.xmpp.srv.scientia.net however.
Do I need additional certs for the canonical hostname?
Or do both hosts need to use the same certificates?
I don't know. I imagine you have a certificate for the host "example.org", and you copy that same file to all the machines.
4) The erlang node name (which is also in the DB) and the Jabber host names, are completely different, right? So I could use other erlang node names, than vhosts served, e.g. as node names the a.xmpp.srv.scientia.net and b.xmpp.srv.scientia.net?
Yes, erlang node name and Jabber domain name are independent topics. In each case put the correct/valid/desired values for each topic.
5) How to configure DNS?
Simply make two SRV records, with the same priorities if I want load balancing or different priorities, if I want my "secondary" server to be just used in case the primary fails?
Yes, that's the general idea.
6) The communication between the two cluster nodes, is it encrypted signed? (with the erlang cookie?)
The erlang cookie is a management tool, used to ensure that no wrong clustering is made in a network with many clusters with many nodes each; it isn't designed as a security measure. Also, the communication between erlang nodes isn't encrypted. And that communication must have low ping, and preferably not closed.
As you can see, the machines that have a cluster of erlang nodes should be in the same LAN, not in the open Internet.