I'm runing ejabberd-2.1-2 on
FreeBSD it05 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:02:08 UTC 2009 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
Suddenly, my /var partition is running out of space.
When I run df -h, i get:
]# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/mirror/gm0s1a 496M 373M 83M 82% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/mirror/gm0s1e 496M 58M 398M 13% /tmp
/dev/mirror/gm0s1f 433G 2.6G 396G 1% /usr
/dev/mirror/gm0s1d 13G 11G 998M 92% /var
procfs 4.0K 4.0K 0B 100% /proc
linprocfs 4.0K 4.0K 0B 100% /usr/compat/linux/proc
When I cd to /var and run a du -hs I get:
# cd /var/
[root@it05 /var]# du -hs *
2.0K account
2.0K agentx
6.0K at
2.0K audit
12K backups
4.0K crash
6.0K cron
168M db
2.0K empty
2.0K games
2.0K heimdal
18K lib
8.8M log
850K mail
4.0K msgs
48K named
4.0K net-snmp
2.0K preserve
54K run
2.0K rwho
3.5M spool
116K tmp
24K yp
So, it seems that something is sucking up all the space but doesn't have a file descriptor on the file system, so du is not finding the file.
Running lsof|grep /var I get:
...
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
beam.smp 28398 ejabberd 10w VREG 0,93 31895 23581 /var/log/ejabberd/erlang.log
beam.smp 28398 ejabberd 11u VREG 0,93 97 47130 /var/spool/ejabberd/LATEST.LOG
beam.smp 28398 ejabberd 12w VREG 0,93 11170768676 23582 /var (/dev/mirror/gm0s1d)
beam.smp 28398 ejabberd 19u VREG 0,93 119220 47140 /var/spool/ejabberd/offline_msg.DAT
beam.smp 28398 ejabberd 20u VREG 0,93 6809 47142 /var/spool/ejabberd/private_storage.DAT
beam.smp 28398 ejabberd 22u VREG 0,93 1282598 47148 /var/spool/ejabberd/pubsub_item.DAT
beam.smp 28398 ejabberd 23u VREG 0,93 5752 47154 /var/spool/ejabberd/vcard.DAT
...
So, it seems that ejabberd is sucking up 10Gb I'm missing on my filesystem, and that space is being reserved into the /var directory.
Is this a bug on ejabberd?
Is there any way to solve this without having to restart the server (not tried yet, as many people is using it....)
Thanks.
This person cross-posted the
This person cross-posted the same question, at the same time, to the ejabberd mailing list. But forgot to mention this fact in both submissions.
See:High filesystem usage
In that discussion, other people noticed that the problem was caused by a mistake in the administrator itself, not a bug in ejabberd.
The issue is offline_msg.DAT.
The issue is offline_msg.DAT. Lately,I have been asked to supply information of logged chat of a certain user. Searching for this in our server, I came accross de said file. However, I just don't how to read it or if there is any plugin to do so as this file has been encrypted, whose algorithm I haven't the foggiest what it is all about.
Can you help me reding that file? Is there any tool to do so? Plugin? Or,maybe, any feature in Ejabberd environment or from command line I missed?
Tks in advance.
Unfortunately I cannot help
Unfortunately I cannot help you with your issue. But I must ask: are you sure you found right topic to post your problem to? I'm afraid it has nothing to do with this topic's issue, and should be posted in a new forum topic.