As I talked about in a previous blog post, I’ve been building my own computers for about 20 years now. Somewhat implied in that blog post but largely glossed over is the fact that when I would build a new computer, I would keep the old one around by slapping Linux on it and using it for server stuff: Elastic, file share, automation, whatever. Mostly, those servers just hung around and worked great, including that one from 15+ years ago that was still somehow chugging along.
In October, an old server I was using as my network file share decided to not turn back on after a reboot (amazingly, it wasn’t the oldest one!). Since that held all my photos, music, and backups, I was not too pleased, although all of that WAS backed up in 2 different places so I didn’t have any real data loss. Fortunately, I was able to move the disks over to my even more ancient server and get access to everything again, even my RAID1 array (thank you mdadm!). This did take a good deal of time, though, and it meant I had to set up some of my services like Kibana again from scratch.
This work got me thinking: should I rely on by-definition old hardware for critical things like my logs, home automation, and network files? And with bespoke configurations too? Not only were they old, but they looked pretty dumb in my server rack in 4U boxes not designed as a server.
What if, I thought, I just created VMs/containers for all of these services, managed them in Terraform and Ansible, and ran them in my own personal cloud using decent hardware? That would mean that I could easily recreate things if I needed to do so, and not have to worry for example if a motherboard went bad. At the same time, I was thinking about messing around with Openstack to learn more about this tech.
This prompted me to combine the efforts: I would buy some refurbished servers, set up Openstack, manage it with Ansible, and create VMs for all my services, once again using Terraform and Ansible. It sounded like a plan.
Getting started with Openstack
Openstack is not for the faint of heart. I came across one comment somewhere that said “With Openstack, it takes six hours of configuration before you can even run anything”. When you are not just setting up Openstack for test purposes by hand, but trying to create Ansible plays to manage it correctly, that 6 hours turns into something like 30. Why didn’t I just use the official Openstack Ansible repo? To be honest, at first glance it looked like a lot, and I didn’t need that complexity. So I decided to create my own simplified Ansible config by taking each step of the installation process and creating the appropriate Ansible plays.
Getting things working was rough. There were some times when the Openstack documentation was wrong, such as incorrect config URLs. I made some mistakes too, such as forgetting the “v2.1” part on an API endpoint, which led to hard to find bugs. Then there was the time that I didn’t realize the VM flavor config was in MB, not GB, and so was incredibly confused when my “8 GB” VM refused to boot; turns out Linux doesn’t run on 8 MB anymore! Troubleshooting networking issues also took up a huge amount of time.
Ansible did come in handy was when I decided to revamp my infrastructure and add another server: by having the config in Ansible, it was easy to scrap it all and start over. That 30 hours of fighting to create Ansible roles for everything meant I could redeploy everything in about an hour, which worked so well I was honestly amazed.
Another useful tool was ChatGPT. After creating a few Ansible roles by hand, I eventually started using it to create new role boilerplate out of the way. AI got me 90% of the way there when I needed a new role for something like Kibana, and then I’d get the last 10% right. AI still has an annoying tendency to do things like hallucinate functions that should exist but don’t, like when ChatGPT and Gemini both hallucinated features of metricbeat that aren’t remotely real. However, AI was very helpful in troubleshooting, able to walk me through commands to run, and then giving suggestions once I pasted the results of those commands back in. It was eerie how it could find solutions to esoteric problems.
Success of some kind
Finally, after defining everything in Ansible and Terraform, I was able to redeploy everything I was running in Openstack VMs. I even used the opportunity to get my EdgeRouter config into Ansible, so I could define firewall rules in Ansible and push to the router. Now, if I want to update the Docker container version of Unifi, or Home Assistant, it’s a simple code change and Ansible run. No more worries about losing a server: I now have compute cattle instead of pets.
Would I do it this way again? Maybe not. I probably should have spent some time with the official Ansible project instead of setting up my own to see if I could have saved some time. I would have had to create Ansible configs for everything else though, so that was necessary work regardless. Openstack itself is a beast with a steep learning curve. However, I probably couldn’t find a better way to run a bunch of VMs (I have 8 currently).
In the end, I’m in a much better spot than I was when I started, even if it was a frustrating journey. If any of my new servers die off, I’ll be able to redeploy fairly easily, and this is great peace of mind.