Hey Internet.
I've been a Linux user since the late 90s, but I'm not generally one to distro hop. I tend to be the sort who finds a thing that works and stick with it because it's familiar. While I've dabbled with other OSes in VMs, I can count on one hand the number of times I've changed my primary OS. I started out with Red Hat Linux back around 1998. At the time, I didn't put much (or really any) thought into which distro I should use. I barely even understood that there were different distros. I went with Red Hat because the copy of "Linux for Dummies" I bought came with an install CD in the back of the book. It was the lowest friction option... which is something of a theme you might notice going forward.
A couple years later, I decided that I should have a look at what other options were out there. After doing a little bit of research I decided that I should give Slackware a try, since I'd heard that it was the most UNIX-ish distribution available at the time. The experience was very different, but after a fair bit of hair-pulling, I managed to get it up and running. That said, this was back in the day where most distributions were somewhat of a pain to get running. I remember well the struggle of trying to figure out how to get my ADSL internet to connect at boot, something I basically take for granted today.
Slackware was my home for a good long while, until one day I decided that maybe automatic dependency management might be a neat thing to have. If you've ever tried to compile GnuCash (or anything else, really) on Slackware, you know what I mean. After some deliberation, I decided to switch to Ubuntu. This was some time in 2008. I only know because I remember the Hardy Heron desktop wallpaper. My reasoning for going with Ubuntu was that it was very popular, and had pretty good community support. I reasoned—correctly, I think—that it would be the easiest option to get running and to fix any errors that might arise over time. It worked out very well for me and I was very happy.
Ubuntu is where I stayed until just recently. At first, I kept up with the latest releases. I would be excited when a new release would drop. There were even release parties in my area that I would attend. Good times were had by all. Since I'm a self-host-all-the-things advocate, after a while I decided to stick with the LTS releases on the server that's hosting all the services I depend on, because not having to upgrade the OS every six months is kind of nice. I'd still use the latest releases on the machine that was my daily driver... for a while. Eventually apathy won out and I stuck with the LTS there as well.
Fast forward to today and Ubuntu is a very different distro than it was back in '08. In the 18.04 release (I think) they introduced snaps, but I mostly ignored them at the time. I didn't care about this newfangled way of installing software. As time progressed, snaps started becoming more and more integrated into the OS until the point where they were almost unavoidable. I'd had trouble with the way snaps were sandboxed, making them unable to access the files in my home directory. Also, my ~/snaps directory was growing out of control on a disk that only had so much space. Eventually, I had to find alternative installation options like Flatpaks and AppImages, but these were really kind of band-aid solutions. The problem I was having was with a core feature of my distro.
When I realized that 22.04 (the latest LTS release) was out, I was faced with a decision: should I upgrade again, or try my hand with another distro? I opted to give Debian a go, since it's the OS that Ubuntu is based on... and it's boring. Debian is not known for being cutting edge, but it is known for being stable. At first, I only updated my main machine to see how difficult the changeover would be. I didn't want to risk screwing up the server, because I kind of depend on it.
The migration of my main machine went (almost) as smoothly as anticipated... though I did have to tether to my phone because the installer didn't have the proprietary drivers for my WiFi card available... a problem I haven't encountered in years with Ubuntu. Nevertheless, it was otherwise uneventful, and I took to Debian like a fish to water. In fact, I even kind of preferred it somewhat. This emboldened me to take the plunge and migrate the server.
Disaster!
To migrate the server, I first decided to be absolutely certain that I had all of my critical data backed up... a good start. The idea was to just nuke the drive entirely, pave over it with the new system, and then restore everything from backup. This was the same thing I had done for the other machine, and all seemed to be going well. Once I'd gotten the system installed and the new SSH keys registered in all the places they needed to be, I started restoring the services I was running. First one up was my Friendica node, because I need my social media fix. Everything restored perfectly from backup, but when I pointed my browser at the page...
500 Internal Server Error
Uh-oh.
Upon querying the console I was told that the database was not present or populated. I double checked. It was. I had the right credentials, but for some reason the app was refusing to acknowledge the existence of the database. After much weeping and wailing and gnashing of teeth, I decided to move on and circle back. Up next was my Gitea server. This was where a bunch of my other backups were stored, so it was a necessary step to get much of the rest up and running again (including my password manager). I restored from backup, ran the service, pointed my browser at it, and...
500 Internal Server Error
Shit.
This one took a bit of poking at, but I came to realize that the problem was that I had misconfigured the SSH tunnel that all of my web services were being run through. If you don't know what an SSH tunnel is, and you do anything internet-related, go learn about them right now. Seriously, they're magic... when they're properly configured that is. Basically, all of my web services are being run on a machine I keep on my desk. They're then forwarded out via SSH tunnel to a cheap VPS for two reasons:
I had accidentally set the SSH tunnel up in a loop that never actually ended up going anywhere. Once I fixed that, my Gitea server started working again. Friendica was still hosed, but it was progress. From there, it was pretty easy to get nearly everything else restored with one other exception.
A while back I re-wrote a program that my father had originally written in 1985 in BASIC to track hockey statistics. It had stopped working properly some time before, due essentially to layers upon layers of backward compatibility finally breaking down. He didn't have time to learn a whole new programming language, so he commissioned me to re-write it for him. After a hard drive crash took out a bunch of his data on the new program, I offered to host it for him on my server, grant him access over an SSH connection, and to manage the backups for him so he didn't have to worry about that happening again.
The data and source restored just fine, but when I went to compile it, I got an error. After some digging, I discovered that the problem was that it required the libghc-ncurses-dev package, which had been been removed from the Debian repositories because it was broken and unmaintanied. Okay, I'll have to figure out what to do about that later.
At this point, I decided to turn my attention back to Friendica. What I had done to back it up was to dump the database and then use borgbackup to back up my webroot, logs, file storage and database dumps. I then restored everything from backup. Apparently, you can't do that. Fortunately, all I had to do was to wipe out the webroot directory and reinstall from scratch. My database backups and file storage were still usable. Once I figured that out, I was able to breathe a sigh of relief.
Finally, I was able to turn my attention back to MTLSTATS (the program I'd re-written for my father). I was worried that I'd have to re-write large chunks of it to make it work again, but I found a simpler solution... even if it was cheating a little. Instead of rewriting it, I re-downloaded the installer for Ubuntu 20.04, spun it up in a VM (with 2 GB of RAM... because Haskell) and compiled it there. I was then able to copy the compiled binary over to the server and it just ran... which was another huge relief. This finally got everything back up and running again.
So, what I expected to be a fairly painless changeover ended up being a far more complicated than anticipated. Since Debian is known for being pretty stable, I hope to never have to distro hop ever again (at least on the server). I'll probably play around with other distros in a VM or on my main machine, but my server's going to stay on Debian. I don't ever want to have to do that again.
Have a good one.