Monitoring your servers with monit

[Shogun 4463]

If you have ever had an incident on your server - cores crashing, filesystem getting full, lag ingame, or ddos attacks- you know that it can turn into detective work to find what is wrong, especially when we don't have a degree in CS. There are some nifty command line monitoring tools (glances, htop, atop come to mind) but when you have several server machines it can get quite frantic and besides - those tools do not offer a lot of flexibility.

 

As we progress in the path to become a skilled server administrator (understanding server as in the machine and the software installed on it rather than in the Metin2 sense), it becomes necessary to use a remote monitoring tool that gives us some visibility on our systems. There is a myriad of paid options featuring flashy animated dashboards out there, but they mostly belong to the cash cow realm of SaaS which I try to stay away of. So today I am going to introduce you to a free monitoring tool -monit- and to its paid console addon, which for some reason the programmers decided to name mmonit, creating unnecessary confusion.

 

You may think that there is already enough documentation out there about popular tools like this - but you're wrong. As well crafted as monit and mmonit are, the documentation is somewhat obscure, particularly when it comes to FreeBSD. So I wanted to write down my experience in getting this tool to work in the best server operating system there is. Yeah, because as much as we like mocking the Ymir programmers, those who worked in the original version were not stupid at all, and the fact that they chose the rock solid FreeBSD -which was a relatively new system back then- is a good proof of that.

 

Part A: Monit

 

So let's get into action and introduce you to Monit. Monit is a very configurable daemon that can check virtually anything on our system -bandwidth, running processes, disk usage...- or even run custom scripts that extend its capabilities, and send us an e-mail to alert when, for example, we are running out of space, or our game server crashes. Nifty isn't it? Here's how an e-mail from monit looks:

Connection failed Service nginx 

	Date:        Sat, 04 Apr 2020 18:05:26
	Action:      alert
	Host:        host.myserver.com
	Description: failed protocol test [HTTP] at [www.myserver.com]:443 [TCP/IP TLS] -- HTTP: Error receiving data -- Operation timed out

Your faithful employee,
Monit

 

Installing monit is easy - just run

pkg install monit

And then let's get our configuration ready.

cp /usr/local/etc/monitrc.sample /usr/local/etc/monitrc

ee /usr/local/etc/monitrc

I will detail here the options we should concern ourselves about - you can leave the rest as default.

 

First is the mail server which we are going to use to submit mail alerts; I entered here my own cPanel mailserver, where we previously whitelisted our IP (Exim Configuration/Access Control). If you don't have your own mail server, you can send the e-mail to sendmail on localhost.

set mailserver mail.myserver.com,               # primary mailserver
                localhost                   # fallback relay

Next goes the e-mail address where we want to receive the alerts:

set alert [email protected] not on { instance, action }

And here we can configure the web interface. Here we set a port, user and password to connect. Simple isn't it?

set httpd port 2812 and
    allow someuser:somepassword

Finally we get to the substance - the monitoring rules. There are plenty of examples in the file, but I will drop my own configurations here as well, including a metin server which is what you're probably interested. This configuration is specific for FreeBSD (12 in my case but should be fine in any modern version)

# Check overall system health
check system $HOST
    if loadavg (1min) per core > 2 for 5 cycles then alert
    if loadavg (5min) per core > 1.5 for 10 cycles then alert
    if cpu usage > 75% for 2 cycles then alert
    if memory usage > 75% for 2 cycles then alert
    if swap usage > 75% for 2 cycles then alert

# Web Server
check process nginx with pidfile /var/run/nginx.pid
   start program = "/usr/local/etc/rc.d/nginx start"
   stop program = "/usr/local/etc/rc.d/nginx stop"
   if failed host www.mysite.com port 80 protocol http and request '/index.html' then alert
   if failed host patch.mysite.com port 80 protocol http and request '/index.php' then alert
check process php-fpm with pidfile /var/run/php-fpm.pid
   start program = "/usr/local/etc/rc.d/php-fpm start"
   stop program = "/usr/local/etc/rc.d/php-fpm stop"
   if children > 30 then alert

# Check network interface and usage. This would warn us about DDoS attacks that rely on
# flooding the connection, or about an overloaded patch server.
check network ix0 with interface ix0
   if failed link then alert
   if saturation > 75% for 2 cycles then alert

# Check filesystem usage - so we don't run out of disk space unexpectedly
check filesystem zroot/ROOT/default with path /
   if space usage > 75% then alert

# Check metin2 game processes, so we will get alerts on a core crash
check process db with pidfile /home/metin2/db/pid.db
  start program = "/home/metin2/db/run.sh"
  stop program = "/home/metin2/db/shutdown.sh"
  mode passive
check process auth with pidfile /home/metin2/auth/pid.auth
  start program = "/home/metin2/auth/run.sh"
  stop program = "/home/metin2/auth/shutdown.sh"
  mode passive
 check process channel1_game1 with pidfile /home/metin2/channel1/game1/pid.game
  start program = "/home/metin2/channel1/game1/run.sh"
  stop program = "/home/metin2/channel1/game1/shutdown.sh"
  mode passive
check process channel1_game2 with pidfile /home/metin2/channel1/game2/pid.game
  start program = "/home/metin2/channel1/game2/run.sh"
  stop program = "/home/metin2/channel1/game2/shutdown.sh"
  mode passive
check process channel2_game1 with pidfile /home/metin2/channel2/game1/pid.game
  start program = "/home/metin2/channel2/game1/run.sh"
  stop program = "/home/metin2/channel2/game1/shutdown.sh"
  mode passive
check process channel2_game2 with pidfile /home/metin2/channel2/game2/pid.game
  start program = "/home/metin2/channel2/game2/run.sh"
  stop program = "/home/metin2/channel2/game2/shutdown.sh"
  mode passive

 

Note that I add "mode passive" because I do not want monit to restart the server if it's down; I only want the alert sent to me. If you want monit to do that, just remove the "mode passive" lines (as I did for nginx and php-fpm)

Also note that channel1_game1 are just names I give them so I can recognize them in the web interface, what matters here is the pid file. Other than that, there are plenty of examples in that file to get you started.

 

Once we are happy with our settings, save and add the following lime to /etc/rc.conf

monit_enable="YES"

 

And now it's showtime:

service monit start

If we set everything properly, we should start getting e-mail alerts on our defined events. It may take some finetuning for you to discover which loads are out of the ordinary for your server and you should be warned about. For example, a simple web server with a 1 Gbps connection having 25% saturation is not normal, but if that web server is being used as a patch server, then you will probably average more than that. You may also use the web interface, though it's pretty basic in monit.

 

This is all regarding monit the free monitoring tool, but for me the most interesting is integrating this system with M/Monit - which is a paid app (60$ for 5 hosts) but with a lifetime license so no monthly payments to worry about. I will get to this soon as it's a perfect solution of you have a bunch of servers to take care of and integrates with FreeBSD and other OS perfectly.

 

Part B: M/Monit

 

As promised, here is the second part of the tutorial concerning M/Monit. M/Monit is a centralized server which collects data from any number of monit hosts and stores it to keep statistics, make alerts more visual. It also allows for stopping and starting services if we set a "Stop" and "Start" parameter for them in monitrc, and you can even show the output of a command from the process list using the "check program" directive in monitrc and hovering over the process name in the M/Monit list.

 

Now it's when things get pretty, with pie charts, responsive interfaces and whatnot. Take a look at this beauty:

 

 

 

 

Now as I said at the beginning there's lots of pretty monitoring interfaces you can buy in the internet and M/Monit is not the prettiest but there's two things that make it stand out:

 

1. It works perfectly with FreeBSD, as it integrates with monit which has been a standard package in FreeBSD for many years.

2. It's not a SaaS cash cow charging you a fortune every month - you pay once and it's yours forever.

 

So first thing you should get yourself a license here: https://mmonit.com/shop/

 

As you can see it's 65€ for up to 5 hosts - lifetime. Those hosts do not need to be FreeBSD, it can be any mix of systems, but I will explain here how to install it on FreeBSD because their documentation, as it happens often, kind of ignores our OS, despite it being supported.

 

While monit should be installed in every server you want to monitor, M/Monit only needs to be installed once, in any server you want (you could even install it in your own PC). Just make sure firewall rules allow traffic between the monit and mmonit instances (monit: port 2812; mmonit: 8080). This M/Monit server should be running a database server - I will assume you have MySQL but SQLite and PostgreSQL are supported too.

 

(credit: I got some of these instructions from a post by SirDice in the official FreeBSD forum)

 

Once you purchased the package extract it (replace the <X.X.X> with your version)

 

tar -C /usr/local -zxvf mmonit-<X.X.X>-freebsd-x64.tar.gz
ln -s /usr/local/mmonit-<X.X.X> /usr/local/mmonit

 

As SirDice explains "The symlink allows you to keep the old versions and switch back in case of problems. ". So whenever you want to install a new version you just need to delete the symlink and then follow these same steps, copying over your configuration to the new folder.

 

Now edit the config:

 

ee /usr/local/mmonit/conf/server.xml

 

The parts we should edit:

 

       <Connector address="x.x.x.x" port="8080" processors="10" />
 

Change that IP for that server's public IP - this is where mmonit will listen for connections.

 

       <Engine name="mmonit" defaultHost="mmonit.mydomain.com" fileCache="10MB">
 

Here we could use the same IP again or, if you want to use a DNS entry instead such as mmonit.mydomain.com, place it here. This is the hostname that we will use to connect to the interface.

 

Now create a database "mmonit" in your MySQL server and a local user like this:

 

CREATE DATABASE mmonit;

CREATE USER 'mmonit'@'localhost' IDENTIFIED BY 'mmonit';

GRANT ALL PRIVILEGES ON mmonit.* TO 'mmonit'@'localhost';

 

and uncomment/edit the following lines in Server.xml:

 

         <Realm url="mysql://mmonit:mmonit@localhost/mmonit"
                  minConnections="5"
                  maxConnections="25"
                  reapConnections="300" />
 

Another line to change here is:

 

           <Host name="mmonit.mydomain.com" appBase="." >

 

Here we should replace it once again with the hostname we defined earlier or the server's IP.

 

Finally at the end of the file you should paste the license information you got when purchasing mmonit.

 

Now let's add a startup script for the service on /usr/local/etc/rc.d/mmonit

#!/bin/sh

# PROVIDE: mmonit
# REQUIRE: DAEMON
# KEYWORD: shutdown

#
# Add the following line to /etc/rc.conf to enable M/Monit
#
# mmonit_enable="YES"

. /etc/rc.subr

name=mmonit
rcvar=mmonit_enable

load_rc_config "$name"

: ${mmonit_enable:="NO"}

command="/usr/local/mmonit/bin/mmonit"
command_args=""
start_precmd="mmonit_checkconfig"
stop_cmd="mmonit_stop"
start_cmd="mmonit_start"

mmonit_checkconfig () {
        echo "Performing sanity check on M/Monit configuration:"
        eval ${command} ${mmonit_flags} -t
}

mmonit_start () {
        echo "Starting M/Monit:"
        eval ${command} ${mmonit_flags} start
}

mmonit_stop () {
        echo "Stopping M/Monit:"
        eval ${command} stop
}

run_rc_command "$1"

 

Remember to chmod 777 this file after you save it, as it is supposed to be an executable.

 

Now and before we run the service, it's time to configure our monit (not mmonit) instances that we set up in the first part to communicate with mmonit. So go back to /usr/local/etc/monitrc in each server where you installed it and uncomment & edit the following lines:

 

set mmonit http://monit:<somepassword>@mmonit.mydomain.com:8080/collector
 

Also in order to issue commands from MMonit to Monit, such as stopping a service remotely, we need to do the following (once again if you don't have your own domain you can replace mmonit.mydomain.com with the M/Monit server IP)

 

set httpd port 2812 and
    allow 127.0.0.1      
    allow mmonit.mydomain.com
    allow monit:<somepassword> 
 

In both cases we replace mmonithost with either the IP address or the domain we set previously for mmonit, and <somepassword> with a secure password - this is used for monit and mmonit to communicate with each other, so we don't need to remember it ourselves.

 

Now let's go back once again to the server where we just installed M/Monit and add this in /etc/rc.conf to auto start mmonit.

 

mmonit_enable="YES" 

 

And start the service with:

 

service mmonit start

 

Now we should be able to browse to the IP or hostname we configured previously.

 

http://mmonit.mydomain.com:8080/

 

The default user and password are admin:swordfish, you should of course change this inmediately from the web interface itself (Admin menu > Users)

 

If everything is done properly, we will get those pretty statistics right in our browser. You can read more about mmonit in the manual:

 

https://mmonit.com/documentation/mmonit_manual.pdf

[Gurgarath 2427]

Your tutorials regarding FreeBSD are always top-notch. This is the kind of tutorial we need and even though I've been using FreeBSD for a solid 10 years now I still sometimes use makeshift scripts or manual inspections to get some of these things done. Which was immensely time-consuming to say the least. I would definitely give it a try.

 

You gave a strong explaination and the example file you provided is a good insight of how powerful this tool can be, I also appreciate the tips you give.

 

1 hour ago, Shogun said:

because as much as we like mocking the Ymir programmers, those who worked in the original version were not stupid at all, and the fact that they chose the rock solid FreeBSD -which was a relatively new system back then- is a good proof of that.

Just to bounce back on this little sentence; because even if I like to gently mock them, I couldn't agree more. The first Ymir team (and a bit of the second team) was solid and did great things with what they had available in the past. For a C++98 game that was made in the early 2000 (or late 90s), some of the things they did were neat and sometimes ingenious. However, the current team which doesn't update the game or stick to the old coding habits from 20 years ago can be mocked a little bit. But I respect their work anyway.

 

Great tuts, congratulations!

[Shogun 4463]

Thanks for the comments. I have added the second part which concerns the central monitoring interface M/Monit.

 

Adding here also this link because there's a couple of interesting ideas - how to get alerted on a specific text found in a log, and how to stress test our server and rules:

 

https://medium.com/@tdebarbora/monitoring-freebsd-with-monit-9ab9493d69bf

Edited April 13, 2020 by Shogun

[ziad 13]

Hello shogun , is there anywhere i can contact you from ?

[Shogun 4463]

I believe this forum allows send private messages