A security checklist for your Metin2 server

[Shogun 4463]

Hi guys,

 

Running a server for a year and a half while a bunch of guys gets paid to destroy it by any means ends up giving you a good insight in what the word security really means. So, after being the target of almost every type of attack possible over the Internet, I think I am ready today to write a quick checklist on how your Metin2 server should be secured from criminal and disruptive behavior. I will divide it into several parts: ingame, website, server, staff.

 

I will try to explain in brief how those attacks are executed, but the focus here is in how to defend from them effectively. I do not want this checklist to serve the purposes of hackers.

 

But be aware; you are never 100% safe - it depends on how much your enemy is willing to work on it. So this should be taken as a bare minimum. And don't forget: the science of destruction evolves constantly. What is considered safe today, might not be so safe tomorrow.

 

Disclaimer: This text reflects 5 years of experience managing a server's technical aspects. I do not do certain things (like quests) for a long time so excuse me for any mistakes I may be making here.

 

 

 

Part 1 - Ingame bugs and exploits

 

This part refers to threats that may realize through the game client or external software that acts through the game client.

 

While often overlooked, sometimes these can be more dangerous to your server than any other type of attack. Keep an eye in your server's economy - the average price of items. If prices for a particular item or all of them change suddenly, there may be some obscure reason behind it. That is one of the reasons why we keep public and private statistics on the average market price of every item at WoM.

 

 

 

One of the most common mistakes - and I commited this myself - when one wants to extensively edit everything is having an item give you more Yang when you NPC it than it costs to buy it from a shop. This gives unlimited Yang to whoever discovers it. There are even public bots which automate the process of buying and selling these items. The effect on the economy is devastating over a sufficient long period, not to mention that it's unfair that some people are infinitely rich and unbalance is never good for any MMORPG. Have your GM double check that no item can be sold for more than it costs.

 

However, this is not the only way that a player can obtain access to unlimited resources. Badly written quests may allow players to obtain their rewards repeatedly by closing their clients while a quest dialog is open. One simple way to avoid this is to make sure that the commands which reward the player are the last in the code block before changing state.

 

 

 

Now going into disruptive behaviour - the most annoying probably are the bugs which cause a server core to crash. The most famous is probably the number_ex bug. This causes several commands which depend on this function such as dice to be exploitable by using certain parameters. This bug is fixed in rev 40k and therefore in any game compiled from the source. If you are using 34k, the best you can do is use iMer's lib which provides several security enhancements. If you are using 2089, there are public diffs that patch this.

 

Another bug exists in the war command in 34k and older which leads to a server crash. You cannot disable this command as it is used by the guild leaders to war other guilds. iMer's Lib takes care of this bug in 34k.

 

Older and less known is the gold drop core crash bug present in 2089 and fixed -I believe- in 34k. Dropping huge amounts of gold or any other item which you can acquire in huge numbers in a map will cause the server to crash. I am not aware of public fixes for this exploit.

 

Another favourite is the kickhack or sync position hack -partially fixed only in 40k- which allows a malicious user to disconnect other players at will. There are even videos of this being done in DE years ago. iMer's lib provides a fix for rev 34k.

 

Finally, we have the long list of cheats which purely seek to gain advantage while playing. There are several client protection tutorials and services around this forum so I won't extend myself here about the subject.

 

 

 

Part 2 - Server and OS

 

Here we talk about the threats that affect your server and Operating System

 

Part 2.1 Choosing and preparing your server

 

There is no doubt at the moment of writing this that the french hosting company OVH provides the most cost-effective anti-ddos solution in the market. While many companies provide DDoS protection services, these cost several times as much as OVH offerings and this increase does not necessarily translate into a better protection. Therefore we will assume that you are renting either a dedicated server at OVH or a VPS at Eterhost as those are the ones I'm familiar with. Other providers or resellers may be configured in a different manner.

 

Dedicated Server at OVH

 

There are two flavors of the Anti-DDoS solution: Basic and Pro. Basic is free when renting a Kimsufi or Soyoustart server (cheap and intermediate OVH brands respectively). What it does is detect attacks directed at your IP address and route your traffic through the mitigation system during the attack. This is not enough to deter a dedicated hacker though - before the attack is detected, your server may be collapsed for a few minutes, and this can be repeated every hour by the attacker, as happened with one of the TEC attacks on WoM about 1 month ago.

 

The PRO version comes with all the OVH proper (also known as professional) line of servers. It adds the ability to route traffic permanently through the mitigation system and provides you with a simple hardware firewall. This is what you should get if serious attacks are a concern, specifically an Enterprise line server (cheapest is at about 100€/month, VAT included)

 

How to activate permanent mitigation on OVH Professional servers:

 

- Open the new manager. If you don't know where this is, open the classic manager and click on "Home" and then below on the "Dedicated" icon.

- Click on IP on the column at the left

- Click the little wheel that appears at the right of your server's IP address

- Select "Mitigation: permanent"

 

How to set up the hardware firewall:

 

- Click the wheel again and select "Activate firewall". Wait around a minute and then reload the page.

- Click the wheel and select "Configure firewall"

- You will be presented with the rules screen. You can add up to 20 rules to deny or allow specific ports on the firewall. Start by adding the allowed ports in the lower numbers and end with a global deny rule. Leaving a field blank in the "Add Rule" dialog will result in a wildcard; so if you leave the IP address field blank, this rule will affect all IPs.

 

Adding an allow port port rule: click on Add Rule; choose rule priority; select protocol (TCP or UDP); select Accept; enter the port number.

Adding a global deny: choose rule priority (must be higher than the allow port rules); select protocol (TCP or UDP); select Deny.

 

 

Here is an example set of rules. We have added the SSH port and the game & auth ports, as well as 3306 so our website can connect to MySQL. You can specify the webserver's IP in the rule to prevent others from connecting to your database, although I personally prefer to do this at the software level with the pf firewall; in any case, port 3306 must not be public.

 

"How do I connect with Navicat then?"

 

Simple, use the SSH Tunnel option. This logs you in to SSH and then connects to the MySQL server as localhost. You must enter your server's SSH login details in the SSH, and your MySQL login details in the General tab, but remember we are connecting to localhost or 127.0.0.1: that's what you must enter in hostname, NOT your server's public address, as MySQL doesn't see us as a remote user anymore.

 

Regarding UDP: UDP is your enemy. Disable it completely as I did in the above ruleset. The official client previously used UDP to check on server status, but it's trivial to override it in python so it always shows "NORM" regardless of the result of servercheck. If you are using 40k or source, the check already uses TCP, but your ports must be lower than 32768 (or the source modified) for it to work.

 

VPS at Eterhost

 

Our VPS are always routed through the mitigation system. Hardware firewall can be activated and configured to your needs for 5€/month extra.

 

Part 2.2 - Hardening SSH

 

First things first: have your OS always updated to the current version.

 

It is often recommended by experts to access your server through a non-administrative account (some operating systems such as CentOS even force you to create one at install), so that's what we will be doing. We will create a restricted user for our game server:

 

adduser game

 

We can leave everything as default except for the login group: enter "wheel" here. Users which are part of the wheel group can use the su command to gain administrator privileges. This allows us to completely block root from remote login.

 

Now it's time to create a SSH key for our new user. This makes bruteforce virtually impossible.

 

Finally, let's prevent login from root, and login without key:

 

ee /etc/ssh/sshd_config

 

Look for the "PermitRootLogin" line and leave it this way, uncommenting if necessary:

 

PermitRootLogin no

 

And next look for the "PasswordAuthentication" setting and edit it so it looks like this:

 

PasswordAuthentication no

 

Now let's restart ssh for changes to take effect:

 

service sshd restart

 

Warning: at this point, you should open a new putty window to test that you can login with the new user and key, and that you can use the su command to gain root privileges. Wrong settings may lock you out of your server.

 

Once everything is working, upload your server files to /home/game. This is the home directory of the user - a bit like "My Documents" in Windows. Make sure you change permissions in the files so the "game" user can read and write them:

 

chown -R game /home/game

 

(this command must be run as root! only root can change someone else's permissions. To switch between the game and root account, you use the su (short for superuser) command:

 

root# su game

Password: (game user's password)

game# su

Password: (root user's password)

root#

 

Or:

 

root# su game

Password: (game user's password)

game# exit

root#

 

Finally, it may be a good idea to install ssh-guard for extra security or change your SSH port if you don't like your logs being spammed by portscans. To change your SSH port, edit the Port line in /etc/ssh/sshd_config and restart the sshd service as indicated above. Remember to open the new port if you are using a firewall or you will lock yourself out of the server.

 

Part 2.3 Software Firewall: pf

 

At this point it's a good idea to add a second line of defense with the pf firewall. First we will create the /etc/pf.conf file following this sample config.

 

ee /etc/pf.conf

 

Make sure you enter the correct interface and all the ports which must be open: SSH and game ports in our case. Neither p2p ports or db core port should be open to the public! Under trusted_hosts, enter the IP address of your web server so it can connect to the database (We assume a typical set up of web server + game / db server here) instead of opening port 3306 to everyone.

 

This firewall provides us with scrubbing and rate limiting capabilities which the hardware firewall doesn't.

 

Next we will enable pf:

 

kldload pf

pfctl -e

 

Check that everything is working fine and then add the following line to /etc/rc.conf to load pf on boot:

 

pf_enable="YES"

[MORTE 76]

spectacular congratulations!

[Yiv 313]

A really helpfully thread if you extend it with the most important security issues!

Thanks for that!

 

Regards

[Metin2 Dev 4666]

#pinned

[Shogun 4463]

I will, I just cant write straight for a long time. Makes me tired.

[Night 363]

I will, I just cant write straight for a long time. Makes me tired.

 

1-this thread is totaly great cool and i love it ^^

2- i kind of feel how writing for too long is

but still keep up the hard work and i think that i will say good night

[Shogun 4463]

Added part 2.1 Choosing and preparing your server

[Mehti 76]

Thanks! I love your posts.

[Shogun 4463]

Added part 2.2 hardening SSH

[Timasu 30]

OFF: Song name?

ON: Will be nice a system like this:

view market price for an item

Edited August 18, 2022 by Metin2 Dev
Core X - External 2 Internal

[Shogun 4463]

Added part 2.3 software firewall

[MORTE 76]

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-pf.html

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-ipfw.html

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-ipf.html

 

add all the help pf fw more blocking?

[daredevil09 5]

What command i need for add permissions for two users? This is for one user: chown -R game /home/game For two?

[Mashkin 16]

I only discovered this checklist now, and I must admit it gives a very good introduction to fundamental security practices.

I would enjoy reading the rest of your explanations on the missing topics.

 

As a (successful) long-time system operator myself, I can nothing but approve this checklist, and urge everyone to educate themselves on systems security.

 

What makes this thread a little unfortunate is the fact that it is required to teach technicians/admins the most basic principles of securing their systems.

Everything written here, except for the game related chapter, is something every sysop must know before they may claim to be a professional.

 

Missing these "skills" is - in my eyes - the primary reason for the enourmous amount of database leaks.

Missing knowledge in general is what I believe to be a major reason for many project failures - set aside "money machines".

[HuNterukh 11]

Thanks a lot!

[Exygo 1045]

OFF: Song name?

ON: Will be nice a system like this:

view market price for an item

OFF: Keep dreaming

 

ON: Great guide Shogun

Edited August 19, 2022 by Metin2 Dev
Core X - External 2 Internal

[mdxyz 238]

thx

[Scooby 25]

thanks

[Dr3Ame3r 32]

https://forums.freebsd.org/threads/4108/

Look here, you have all the information you need.

[Mind Rapist 178]

#Respect to @Shogun and everyone who shares free security solutions for servers out there. Security is not a priviledge, it's something nessecary and sharing it is what makes us human beings...

Thanks so much for this guide I appreciate it a lot 

[4XCIRCO 47]

On 8/28/2014 at 1:02 AM, Shogun said:

Hi guys,

 

Running a server for a year and a half while a bunch of guys gets paid to destroy it by any means ends up giving you a good insight in what the word security really means. So, after being the target of almost every type of attack possible over the Internet, I think I am ready today to write a quick checklist on how your Metin2 server should be secured from criminal and disruptive behavior. I will divide it into several parts: ingame, website, server, staff.

 

I will try to explain in brief how those attacks are executed, but the focus here is in how to defend from them effectively. I do not want this checklist to serve the purposes of hackers.

 

But be aware; you are never 100% safe - it depends on how much your enemy is willing to work on it. So this should be taken as a bare minimum. And don't forget: the science of destruction evolves constantly. What is considered safe today, might not be so safe tomorrow.

 

Disclaimer: This text reflects 5 years of experience managing a server's technical aspects. I do not do certain things (like quests) for a long time so excuse me for any mistakes I may be making here.

 

 

 

Part 1 - Ingame bugs and exploits

 

This part refers to threats that may realize through the game client or external software that acts through the game client.

 

While often overlooked, sometimes these can be more dangerous to your server than any other type of attack. Keep an eye in your server's economy - the average price of items. If prices for a particular item or all of them change suddenly, there may be some obscure reason behind it. That is one of the reasons why we keep public and private statistics on the average market price of every item at WoM.

 

 

 

One of the most common mistakes - and I commited this myself - when one wants to extensively edit everything is having an item give you more Yang when you NPC it than it costs to buy it from a shop. This gives unlimited Yang to whoever discovers it. There are even public bots which automate the process of buying and selling these items. The effect on the economy is devastating over a sufficient long period, not to mention that it's unfair that some people are infinitely rich and unbalance is never good for any MMORPG. Have your GM double check that no item can be sold for more than it costs.

 

However, this is not the only way that a player can obtain access to unlimited resources. Badly written quests may allow players to obtain their rewards repeatedly by closing their clients while a quest dialog is open. One simple way to avoid this is to make sure that the commands which reward the player are the last in the code block before changing state.

 

 

 

Now going into disruptive behaviour - the most annoying probably are the bugs which cause a server core to crash. The most famous is probably the number_ex bug. This causes several commands which depend on this function such as dice to be exploitable by using certain parameters. This bug is fixed in rev 40k and therefore in any game compiled from the source. If you are using 34k, the best you can do is use iMer's lib which provides several security enhancements. If you are using 2089, there are public diffs that patch this.

 

Another bug exists in the war command in 34k and older which leads to a server crash. You cannot disable this command as it is used by the guild leaders to war other guilds. iMer's Lib takes care of this bug in 34k.

 

Older and less known is the gold drop core crash bug present in 2089 and fixed -I believe- in 34k. Dropping huge amounts of gold or any other item which you can acquire in huge numbers in a map will cause the server to crash. I am not aware of public fixes for this exploit.

 

Another favourite is the kickhack or sync position hack -partially fixed only in 40k- which allows a malicious user to disconnect other players at will. There are even videos of this being done in DE years ago. iMer's lib provides a fix for rev 34k.

 

Finally, we have the long list of cheats which purely seek to gain advantage while playing. There are several client protection tutorials and services around this forum so I won't extend myself here about the subject.

 

 

 

Part 2 - Server and OS

 

Here we talk about the threats that affect your server and Operating System

 

Part 2.1 Choosing and preparing your server

 

There is no doubt at the moment of writing this that the french hosting company OVH provides the most cost-effective anti-ddos solution in the market. While many companies provide DDoS protection services, these cost several times as much as OVH offerings and this increase does not necessarily translate into a better protection. Therefore we will assume that you are renting either a dedicated server at OVH or a VPS at Eterhost as those are the ones I'm familiar with. Other providers or resellers may be configured in a different manner.

 

Dedicated Server at OVH

 

There are two flavors of the Anti-DDoS solution: Basic and Pro. Basic is free when renting a Kimsufi or Soyoustart server (cheap and intermediate OVH brands respectively). What it does is detect attacks directed at your IP address and route your traffic through the mitigation system during the attack. This is not enough to deter a dedicated hacker though - before the attack is detected, your server may be collapsed for a few minutes, and this can be repeated every hour by the attacker, as happened with one of the TEC attacks on WoM about 1 month ago.

 

The PRO version comes with all the OVH proper (also known as professional) line of servers. It adds the ability to route traffic permanently through the mitigation system and provides you with a simple hardware firewall. This is what you should get if serious attacks are a concern, specifically an Enterprise line server (cheapest is at about 100€/month, VAT included)

 

How to activate permanent mitigation on OVH Professional servers:

 

- Open the new manager. If you don't know where this is, open the classic manager and click on "Home" and then below on the "Dedicated" icon.

- Click on IP on the column at the left

- Click the little wheel that appears at the right of your server's IP address

- Select "Mitigation: permanent"

 

How to set up the hardware firewall:

 

- Click the wheel again and select "Activate firewall". Wait around a minute and then reload the page.

- Click the wheel and select "Configure firewall"

- You will be presented with the rules screen. You can add up to 20 rules to deny or allow specific ports on the firewall. Start by adding the allowed ports in the lower numbers and end with a global deny rule. Leaving a field blank in the "Add Rule" dialog will result in a wildcard; so if you leave the IP address field blank, this rule will affect all IPs.

 

Adding an allow port port rule: click on Add Rule; choose rule priority; select protocol (TCP or UDP); select Accept; enter the port number.

Adding a global deny: choose rule priority (must be higher than the allow port rules); select protocol (TCP or UDP); select Deny.

 

 

Here is an example set of rules. We have added the SSH port and the game & auth ports, as well as 3306 so our website can connect to MySQL. You can specify the webserver's IP in the rule to prevent others from connecting to your database, although I personally prefer to do this at the software level with the pf firewall; in any case, port 3306 must not be public.

 

"How do I connect with Navicat then?"

 

Simple, use the SSH Tunnel option. This logs you in to SSH and then connects to the MySQL server as localhost. You must enter your server's SSH login details in the SSH, and your MySQL login details in the General tab, but remember we are connecting to localhost or 127.0.0.1: that's what you must enter in hostname, NOT your server's public address, as MySQL doesn't see us as a remote user anymore.

 

Regarding UDP: UDP is your enemy. Disable it completely as I did in the above ruleset. The official client previously used UDP to check on server status, but it's trivial to override it in python so it always shows "NORM" regardless of the result of servercheck. If you are using 40k or source, the check already uses TCP, but your ports must be lower than 32768 (or the source modified) for it to work.

 

VPS at Eterhost

 

Our VPS are always routed through the mitigation system. Hardware firewall can be activated and configured to your needs for 5€/month extra.

 

Part 2.2 - Hardening SSH

 

First things first: have your OS always updated to the current version.

 

It is often recommended by experts to access your server through a non-administrative account (some operating systems such as CentOS even force you to create one at install), so that's what we will be doing. We will create a restricted user for our game server:

 

adduser game

 

We can leave everything as default except for the login group: enter "wheel" here. Users which are part of the wheel group can use the su command to gain administrator privileges. This allows us to completely block root from remote login.

 

Now it's time to create a SSH key for our new user. This makes bruteforce virtually impossible.

 

Finally, let's prevent login from root, and login without key:

 

ee /etc/ssh/sshd_config

 

Look for the "PermitRootLogin" line and leave it this way, uncommenting if necessary:

 

PermitRootLogin no

 

And next look for the "PasswordAuthentication" setting and edit it so it looks like this:

 

PasswordAuthentication no

 

Now let's restart ssh for changes to take effect:

 

service sshd restart

 

Warning: at this point, you should open a new putty window to test that you can login with the new user and key, and that you can use the su command to gain root privileges. Wrong settings may lock you out of your server.

 

Once everything is working, upload your server files to /home/game. This is the home directory of the user - a bit like "My Documents" in Windows. Make sure you change permissions in the files so the "game" user can read and write them:

 

chown -R game /home/game

 

(this command must be run as root! only root can change someone else's permissions. To switch between the game and root account, you use the su (short for superuser) command:

 

root# su game

Password: (game user's password)

game# su

Password: (root user's password)

root#

 

Or:

 

root# su game

Password: (game user's password)

game# exit

root#

 

Finally, it may be a good idea to install ssh-guard for extra security or change your SSH port if you don't like your logs being spammed by portscans. To change your SSH port, edit the Port line in /etc/ssh/sshd_config and restart the sshd service as indicated above. Remember to open the new port if you are using a firewall or you will lock yourself out of the server.

 

Part 2.3 Software Firewall: pf

 

At this point it's a good idea to add a second line of defense with the pf firewall. First we will create the /etc/pf.conf file following this sample config.

 

ee /etc/pf.conf

 

Make sure you enter the correct interface and all the ports which must be open: SSH and game ports in our case. Neither p2p ports or db core port should be open to the public! Under trusted_hosts, enter the IP address of your web server so it can connect to the database (We assume a typical set up of web server + game / db server here) instead of opening port 3306 to everyone.

 

This firewall provides us with scrubbing and rate limiting capabilities which the hardware firewall doesn't.

 

Next we will enable pf:

 

kldload pf

pfctl -e

 

Check that everything is working fine and then add the following line to /etc/rc.conf to load pf on boot:

 

pf_enable="YES"

Top notch. Love to see security being openly mentioned and shared with the community!