Hey Internet.
Back in September or October of last year I decided I wanted to play some Minetest. For those who don't know, Minetest is a free and open source software (FOSS) game engine that started out as an attempt to re-create Minecraft, which I quite enjoyed playing back in the day. Given my general enthusiasm for FOSS, I decided it was worth having a look at. The engine comes with a really basic game simply called "Minetest Game". Out of the box, it's really rather boring.
The thing that makes Minetest interesting though is that it's designed from the ground up to be modded. Unlike Minecraft, mods are officially supported, instead of having to be built in an ad-hoc way that breaks with every update. They even provide a built-in directory of mods and even entire games for the engine that can be installed from a point-and-click menu, so after spicing the game up with a dozen or so mods, I was hooked.
Naturally, the next step was to set up my own Minetest server so I could play with family and friends. I already had a cheap VPS that I was using (to host this blog, for instance) so I figured setting up a Minetest server would be a walk in the park. Little did I realize the wild ride that I was in for.
Since I'm very budget conscious (some would even say cheap) I only have a single VPS on the cheapest plan Linode has to offer... which isn't very powerful. Despite its limitations, it "hosts" pretty much all the internet services I run... way more than it ought to be able to, in fact. That's because it's really only being used as a glorified router. Most of the heavy lifting is being done by an old laptop that sits in the closet of my office. It's connected via a WireGuard VPN to the VPS and the VPS simply port forwards to it. At the time, this was primarily done through SSH tunnels.
Unfortunately, since Minetest uses the UDP protocol instead of TCP (presumably for performance reasons) an SSH tunnel wasn't going to cut it. SSH tunnels only work for TCP connections. After a while of fumbling around trying to figure out how to forward a UDP port, I grew frustrated and decided to see if it was possible to run the server directly on the VPS. Much to my surprise, this actually worked... on a box with a measly 1GB of RAM! This, I have to admit, is truly a testament to how well written the Minetest codebase is. I really didn't expect that.
For several months, it just ran like this, and since it was just me on the server (and occasionally Katy) it ran pretty decently. Life was good. Eventually, I decided to have my server announce itself to the default server list so others would know it existed, and possibly join. I'd get the occasional visitor, but they typically only stuck around for a few minutes, got bored, and moved on. Then, out of nowhere, the unthinkable happened: for some reason I still don't understand, my server got somewhat popular.
I actually had a regular user who kept coming back... then two... three... Within a few months, it was not uncommon for a handful of users to be simultaneously connected, which was great, but the server started straining to keep up. It was noticeable, but still playable.
There was also another problem: griefers. It was a problem I'd anticipated, but until recently had failed to materialize in any significant capacity. Occasionally, I'd have to re-plant some trees and crops, or have to re-build a fence, but nothing major. Besides, I backed the server up pretty much every weekday, so in a worst-case-scenario, I could always restore from a previous state. Then, one Monday, I signed in to find everything on fire and signs with racist profanity put up everywhere.
There was no way for me to fix the damage; I had to restore from backup. Unfortunately, the last backup I'd made was on Friday morning. Two of my regulars had spent the whole weekend building some pretty cool stuff and all that work was simply lost. There was nothing I could do about it. Needless to say, they were not happy. Shortly thereafter, I added a protection mod so that I could keep certain key areas safe from harm. I had really hoped I wouldn't need to do that, but there wasn't much of an alternative.
Meanwhile, the server kept growing in popularity and my poor little bargain basement VPS just couldn't keep up. If there were about a half dozen people or more simultaneously connected, the lag was simply unbearable. Multiple times a day it would just get so overloaded that it would crash, kicking everyone out and restart itself. People started complaining (and rightly so). Something had to give. I have to admit, the idea of just shuttering the server occurred to me, but by this point I had a number of people who regularly spent several hours a day playing on it, and I didn't want to do that to them.
I decided it was time to finally figure out this UDP port forwarding nonsense. Basically, I was able to set iptables up so that UDP packets from the outside world were being forwarded to the machine I wanted to use for the server in the first place, but the server's replies to those clients were simply not being delivered. After a bit of mucking around, I discovered that this was because I'd set WireGuard up to only handle traffic for the 192.168.3.0/24 IP range (my VPN's internal IP range). I didn't want to burden it with handling all my internet traffic. That seemed excessive. (I was not wrong about this, but more on that later.)
It turned out that reconfiguring the VPN to handle all that machine's internet traffic did the trick... for a while anyway, but little did I realize I'd just activated a ticking time bomb. This machine handles a lot more than just my Minetest server. It also hosts this web site, my Friendica node, and a bunch of other things too. Friendica, for instance, sends out notification emails—quite a few, in fact—and unbeknownst to me, my VPS was filtering outbound SMTP connections. Instead of being delivered, these emails were simply piling up in my postfix queue. This went on for maybe a week or so before I noticed.
Once I figured out what was happening... and how to fix it (by setting up a custom routing rule to handle traffic to the mail server) postfix tried to deliver the entire backlog at once. Needless to say, this triggered a spam filter and got me rate limited by my mail provider. Nevertheless, six hours (and a flushed postfix queue) later, I was back in business.
Still, there was one problem remaining. Remember how I said I didn't think it was a good idea to route all that machine's traffic through the VPS? Turns out I wasn't wrong. As the amount of traffic being put through it increased, it started once again having trouble keeping up. At the time of writing this, that's still a problem I'm trying to figure out how to fully mitigate.
My first idea was to see if I could only route UDP traffic through the VPN, but since routing happens at the IP level, there really wasn't a good way to do that. Another idea I had was to create a new WireGuard interface that would only be visible/available to the dedicated user I'd created to run the Minetest server, but it seems the only way to do that involves essentially running it in a VM, which is a little on the ridiculous side.
I then came up with an alternative solution, though it's admittedly a bit of an ugly hack. It turns out that Minetest only supports IPv4 traffic(!?) It also turns out that the bulk of the traffic being routed through the VPN is IPv6 traffic, so I was able to stop routing all the IPv6 traffic through it. This of course is not ideal, but it's a Good Enough for Now™ solution.
After several days of trying to fix this, I've come to the conclusion that the best solution is to just... get a better VPS. I've been meaning to switch from Linode to Hetzner anyway; I've just been too lazy to do it. It's going to be kind of a pain, but I'm sure it'll be worth it in the end.
Oh, I almost forgot, if you want to have a look at the server I've built, the details are here.
Have a good one.