Load balancing with Nginx

[Shogun 4465]

So I was checking out Aeldra's Discord and apparently their patcher was slow as a snail, which reminded me of the opening of WoM3 back in the day.

The typical OVH dedicated server has a bandwidth of 100 Mbps upstream if I remember well, although you can buy more bandwidth (which costs as much as the server itself) and some come with 1 Gbps. Anyway that's a gigaBIT per second which makes around 150 megabytes per second. Pretty good but not enough for a big opening thing where you have hundreds of users expecting to download at 4 Mbps each.

NGINX (the one you pay for) includes some load balancing mechanism but we can emulate it with the free one. We are in fact sorta randomly distributing people among a number of servers.

So here is how you can just throw money at the problem (if you have this problem you're probably gonna get rich anyway) and just rent more servers or just use a bunch of VPS you have lying around. I'm not going to make lengthy explanations here if u need help you know where to find me and my paypal.

So we have our url let's say patch.wom2.org pointing to our main webserver, Here's where the magic happens (http context)

split_clients "${remote_addr}" $destination {
       50%   alpha.patch.wom2.org:8080;
       30%   bravo.patch.wom2.org:8080;
       20%   charlie.patch.wom2.org:8080;
 }

Here we have three subdomains pointing at three different servers (any place where u can install nginx will do, you can measure speed at the endpoint with nethogs for example to see which is slower and reduce the percent of requests that are sent there). The OG web server can serve files too (here it's alpha). Do not use Linux if you can avoid it. And do NOT use Apache. FreeBSD is the king when it comes to streaming sry Linux fanboys.

The three servers must of course have identical copies of your files (use rsync when updating patches) and the same nginx configuration. Here's the config for the OG server which redirects the user to the previously chosen subdomain when downloading from the pack directory (server context obviously)

server {
        listen 51.84.214.58:80;
        server_name patch.wom2.org;

        root  /home/www/patch.wom2.org;


        location /1.1.1.1/ {
                 log_not_found on;
                 return 302 http://$destination$request_uri;
        }

}

Finally here's the config of one of our load balancing server, which in its root folder contains the contents of 1.1.1.1, in fact alpha.patch and patch are in the same folder.

 server {
        listen  51.84.214.58:8080 sndbuf=32k;
        server_name alpha.patch.wom2.org;
        root  /home/www/patch.wom2.org;

        location / {
                 limit_rate 4096k;
                 if_modified_since off;
                 expires epoch;
         }
}

As you can see I limited speed to 4 Mbps to avoid people with a big pipe taking all the bandwidth.

Remember no Cloudflare here, CF is not for file serving.

Edited July 29, 2022 by Shogun

[Speachless 748]

And for the people who don't want to deal with this tech part, maybe for some of them it's complicated they can have let's say main.mydomain.com which at every patchload randomly redirects to s1.mydomain.com s2.mydomain.com who are hosted on different hosts. (you can use cloudflare to update the ips in short time), but without activating the cloudflare dns otherwise you will have that patcher error (and as far as i know cloudflare doesnt allow high bandwidth usage). Also patchers do not require high ram / high cpu hosts. just average normal hosts. it's all about internet here. Want to redirect also based on their location ? You can activate the cloudflare dns on the main subdomain where the redirect happens, and use the $_SERVER["HTTP_CF_IPCOUNTRY"] to get their language.

[Shogun 4465]

they can have let's say main.mydomain.com which at every patchload randomly redirects to s1.mydomain.com s2.mydomain.com who are hosted on different hosts

That's literally what I just described? I'm not sure what you mean with your message.

[Speachless 748]

On 9/10/2022 at 6:42 AM, Shogun said:

they can have let's say main.mydomain.com which at every patchload randomly redirects to s1.mydomain.com s2.mydomain.com who are hosted on different hosts

That's literally what I just described? I'm not sure what you mean with your message.

All i wanted to say is you can do it with simple redirects even in cpanel and how they can do it even for different locations. Europe, Asia, America whatever. Your post is about nginx/freebsd, but this load balancing can be done with any type of webhost.

[Shogun 4465]

Care to elaborate on your answer? Nginx is not "a webhost" and neither is cPanel. Cloudflare does not do load balancing in free accounts, certainly not with "simple redirects" and "quickly changing the IP" (what kind of insanity is that). Neither does sending people to a redirect based on their country qualify as "load balancing", unless the amount of users per country is predictable but still, it's a weird idea, unless we're talking about geographically better located hosts, which still isn't load balancing.

Bottom line this is a developer forum, I would expect "people who don't want to deal with this tech part" to find their entertainment elsewhere.

[Amun 1816]

Pretty cool stuff!

However, I would've liked to see some reasons as to why I would want to set up "a bunch of servers" to serve the data to my players when I could just use a bucket on google cloud or something like Amazon's object storage?

I mean, they only cost around 2$  1$/100GB(0.01$/GB) and they're pretty easy to set up.

 

Any thoughts/feedback on this?

Edited September 10, 2022 by Amun
Updated pricing, free retrieval, but 0.01$ network costs/GB

[Shogun 4465]

1 hour ago, Amun said:

Pretty cool stuff!

However, I would've liked to see some reasons as to why I would want to set up "a bunch of servers" to serve the data to my players when I could just use a bucket on google cloud or something like Amazon's object storage?

I mean, they only cost around 2$  1$/100GB(0.01$/GB) and they're pretty easy to set up.

 

Any thoughts/feedback on this?

I assume it's a bunch of servers you already paying for because you're using them for something else, not specifically rented for this.

Naturally I haven't used "cloud storage" because that's marketed to developers since as a systems administrator it doesn't make sense to pay for services you can't fully control, but if you are serving files frequently to a large userbase that bill can become non-trivial.

[Amun 1816]

Spoiler

  

27 minutes ago, Shogun said:

I assume it's a bunch of servers you already paying for because you're using them for something else, not specifically rented for this.

Naturally I haven't used "cloud storage" because that's marketed to developers since as a systems administrator it doesn't make sense to pay for services you can't fully control, but if you are serving files frequently to a large userbase that bill can become non-trivial.

 

Please don't take this the wrong way, I'm just asking because you had a server for a fairly long time, so you have some insight into what's needed to maintain it properly and, most likely, stumbled upon problems(/situations?) that most developers/owners may not take into consideration(or even think about). 

Going back to our storage discussion, for me it doesn't really make sense to pay for any of them if you're not in the top 1% of the servers(the ones that actually need it). If you have some servers lying around(which most people don't), then yeah, whatever, you do you.

IMO, allowing people to download the full client from the main server is a mistake, considering that Google drive and Mega have amazing download speeds and pretty big quotas. Using it for your patcher, yeah, that's fine, most files in one's client will never change, so patching the scripts/locals/exe isn't going to be that big of a deal when you barely make an update a week.

 

TLDR: Don't spend money on unnecessary infrastructure, use GDrive/Mega, and don't offer direct download links from your main server, save the bandwidth for patches and serving the actual website(which is why you got the server in the first place).

 

Yeah, so.. my rant is done, hopefully someone will find it useful.

Have a great day,

- Amun

[Speachless 748]

On 9/10/2022 at 10:37 AM, Shogun said:

Care to elaborate on your answer? Nginx is not "a webhost" and neither is cPanel. Cloudflare does not do load balancing in free accounts, certainly not with "simple redirects" and "quickly changing the IP" (what kind of insanity is that). Neither does sending people to a redirect based on their country qualify as "load balancing", unless the amount of users per country is predictable but still, it's a weird idea, unless we're talking about geographically better located hosts, which still isn't load balancing.

Bottom line this is a developer forum, I would expect "people who don't want to deal with this tech part" to find their entertainment elsewhere.

That's far from what i wanted to say. Simply said: All your complicated tutorial can be easily done by pointing the patcher to domain.com which then he redirects you to one of his domains s1.mydomain.com s2.mydomain.com who are pointing to different ips(hosts). In case one goes down, using cloudflare you can change the ip it points in seconds without the traffic to actually pass through cloudflare's ips. Is that hard to understand ? Jesus christ. And can be done with any webhost, cpanel, plesk whatever is there. You just need a simple redirect not using freebsd or centos. Sorry, but cpanel is still superior to any manual workaround. The time you spend configuring a centos server or whatever, or as you say freebsd, is not worth for me (though i used to have patcher on centos vps). I think everyone is free to chose what kind of solution they want. Why you get triggered ? Ps: Romanian cpanel webhosts, especially cloud ones are cheap, come with high internet speed, no bandwidth limit and with good cpu/ram/space resources. I am aware other countries doesn't have this options, but anyone can rent a romanian host and see actually what a good speed they have all over Europe. If not, a vps with freebsd, centos or whatever is cheaper and the only option. I won't give any other answers here.

[Shogun 4465]

On 9/10/2022 at 6:53 PM, Amun said:

  Hide contents

  

 

Please don't take this the wrong way, I'm just asking because you had a server for a fairly long time, so you have some insight into what's needed to maintain it properly and, most likely, stumbled upon problems(/situations?) that most developers/owners may not take into consideration(or even think about). 

Going back to our storage discussion, for me it doesn't really make sense to pay for any of them if you're not in the top 1% of the servers(the ones that actually need it). If you have some servers lying around(which most people don't), then yeah, whatever, you do you.

IMO, allowing people to download the full client from the main server is a mistake, considering that Google drive and Mega have amazing download speeds and pretty big quotas. Using it for your patcher, yeah, that's fine, most files in one's client will never change, so patching the scripts/locals/exe isn't going to be that big of a deal when you barely make an update a week.

 

TLDR: Don't spend money on unnecessary infrastructure, use GDrive/Mega, and don't offer direct download links from your main server, save the bandwidth for patches and serving the actual website(which is why you got the server in the first place).

 

Yeah, so.. my rant is done, hopefully someone will find it useful.

Have a great day,

- Amun

People can report stuff you upload to Google Drive or Mega claiming it's infringing their copyright. Free stuff isn't really free, it comes at the cost of handing over control over your data and your privacy.

Anyhow if you don't have a big server you won't have thousands of people trying to download your files so you don't need load balancing. You won't have haters reporting your client to MEGA or Google and getting you suspended either. On the other hand, I do consider Amazon buckets and SaaS in general to be unnecesary expenses when there is a world of software you can host yourself for free in a simple Proxmox server.

This is the hidden content, please

• Sign In
• or
• Sign Up

 

 

 

 

On 9/10/2022 at 7:27 PM, Speachless said:

That's far from what i wanted to say. Simply said: All your complicated tutorial can be easily done by pointing the patcher to domain.com which then he redirects you to one of his domains s1.mydomain.com s2.mydomain.com who are pointing to different ips(hosts). In case one goes down, using cloudflare you can change the ip it points in seconds without the traffic to actually pass through cloudflare's ips. Is that hard to understand ? Jesus christ. And can be done with any webhost, cpanel, plesk whatever is there. You just need a simple redirect not using freebsd or centos. Sorry, but cpanel is still superior to any manual workaround. The time you spend configuring a centos server or whatever, or as you say freebsd, is not worth for me (though i used to have patcher on centos vps). I think everyone is free to chose what kind of solution they want. Why you get triggered ? Ps: Romanian cpanel webhosts, especially cloud ones are cheap, come with high internet speed, no bandwidth limit and with good cpu/ram/space resources. I am aware other countries doesn't have this options, but anyone can rent a romanian host and see actually what a good speed they have all over Europe. If not, a vps with freebsd, centos or whatever is cheaper and the only option. I won't give any other answers here.

I am not a developer like most people here are are, but a systems administrator. "The time you spend configuring a centos server" -as you describe it- is not a burden like it's for you, it's literally my job. And I have plenty of work thanks to people like you who think because they know a programming language or two, they know anything about the systems that lie under them; I have some news for you: you don't. So it's great that Cloudflare caters to people like you and make a business out of it, some of us like to have control over our infrastructure and pay only for the metal it's running in. Also your thinking that changing a DNS in CF when your host goes down is cutting edge technology is so ridiculous I won't even comment of it.

Romanian hosts? I think you are confusing your home internet -which is indeed fastest in the EU, and second cheapest after Poland- with servers. It's okay, you're just a programmer after all. As a matter of fact, Romanian datacenters have poor connectivity when compared to other EU countries. Of course, it's still much better than, say, Russia or Morocco, but at European level it's nothing to brag about, trust me.

 

[Speachless 748]

You don't need to attack me with that because my intention here was not to attack you. Wanted just to say this is possible to make with cpanel for everyone who wants to do it with cpanel. Just because you spend time to configure a centos or bsd vps in a complicated way won't make the download faster or the vps better protected to attacks and i don't want to say you shouldn't do it just because cpanel exists, i used to configure vps like you explained here in the past. Contabo, ovh sells vps with weak internet bandwidth something like 100 mbps, 250 mbps. Speaking from my experience when i visited Spain i had really good download speed from my romanian hosted patcher. I'm not saying romanian's the best, but for me, after trying so many other countries romania provided the best download speeds for the patcher, but not the most stable internet for a hosted server.

"So it's great that Cloudflare caters to people like you and make a business out of it, some of us like to have control over our infrastructure and pay only for the metal it's running in. "  = i am not paying anything to cloudflare. It simply allows me to change fast to which ip a domain points, which is great.

"Also your thinking that changing a DNS in CF when your host goes down is cutting edge technology is so ridiculous I won't even comment of it."  = You should think twice. There will be 2-3 momments in years in which it will for any reason go down (host issue, strong ddos attack), remember the fire that happened to OVH where lots of vps were lost ? So if my vps/host goes down shouldn't i be able to change the ip the domain points ? with CF i can do it in seconds.

Let's also put a strong point on high bandwidth servers too and where we can find them at a low price

And i shouldn't have started this subject from the first so let's end it because i'm not here to say you do it wrong or whatever. You better ask a moderator to clear the comments.

Example:

In your patcher config xml, make it point to this torrent_config.php file

Then this is the website part

Or with redirect:

<?php
$rand = rand(1,3);

if ($rand == 1)
	header("Location: http://patcher.yourwebsite.com/torrent.config.xml");
elseif ($rand == 2)
	header("Location: http://patcher.yourwebsite.com/torrent.config2.xml");
else
	header("Location: http://patcher.yourwebsite.com/torrent.config3.xml");
?>

If people don't know what torrent.config.xml contains:

I think it's better i am displaying directly the xml content rather than redirecting to it. Anyway, this is how you can easily do it in php. Tested and working fine, however your method is very good in having the ability to create your own firewall rules. On that i can agree.

Edited October 10, 2022 by Speachless
Core X - External 2 Internal