Clouds

The €154, 5-node, HA, hyper-converged Proxmox private cloud on Hetzner

For the past few years I’ve been paying around €60/month at Google Cloud to host the equivalent of 4 cores and 8 GB of RAM in total on all my instances. Recently I converted my home i7-3770 to an Proxmox based server and found it super simple to work with it. Through a combination of No-IP, DNS CNAMEs and an HAproxy instance forwarded through my router I was able to get many applications easily installed (and backed-up to S3 through TurnKeyLinux TKLBAM/backup which runs Duplicity) around every single day. So much for complexity as in about 3 days I had pretty much everything up (Nexus, Go.CD and agents, this blog, Mattermost, Nextcloud, etc.)

In the past 3 days I rediscovered Hetzner. I knew them for a long time but I wasn’t so keen on renting “dedis” (dedicated servers). Up until I discovered their server auction going around the €30 per i7-3770 with 2x3TB of HDD and 32GB of RAM.

Initially I just fooled around and played around with 1 machine and the “installimage” script, trying out to see if it’s easy to set-up Debian 10 and PVE. It went smooth. Then I explored the networking part, trying to see if I could get an private subnet on the same VM to be routed in the so-called “single IP, routed configuration” that Proxmox suggests.

Then I ordered two more servers, made a vSwitch between them, added the same 192.168.0.0/16 network to all of them (in the idea that they run LXCs or VMs with static IPs and I route the traffic to the proper services). Then I pinged from host “pve1” to a container on some other host and it worked amazingly well. I also put Proxmox in Cluster configuration and installed Ceph on the second non-used disk (I’ve put LVM on the first disk and Ceph on the others).

Given 3 servers I wanted 5 to be sure that any cloud I make will survive any downtime of any single node. Now I’m waiting the weekend out to have the guys at Hetzner approve and install my 2 extra servers. I’m anxiously waiting.

Back to the DIY …

I know it’s not enterprise hardware and the “servers” I’ve bought are desktop-grade hardware but I have two main motives for why I don’t care that much:

  • my home PC has 7 years and going strong, in a desktop configuration;
  • it will be mostly a GitOps deployment (configuration to be managed by code);
  • data’s mostly directly written to S3 (eg. Mattermost, Nextcloud) with the apps only acting as proxy) or backed-up with TKLBAM, for which I can survive a few hours downtime until I reset everything;
  • 5x physical different nodes over multiple DCs and locations is quite strong, even if there’s that 1Gpbs limit (which is actually 100MB/s and I ain’t doing Big Data or HPC on a 5x node cluster, seriously);
  • add-up the resources, resulting in a 40-core cloud, at 160GB of RAM given 15TB of Ceph storage (and an extra 10TB of LVM-thin);
    • for only €154 a month which adds up to €1848 per year;
    • versus the €30k+ that a 40-core mammoth server-grade machine would cost;
      • by comparison I would reach the same price in 15 years;
  • lastly, I can continue watching the server auction of Hetzner and if there’s newer hardware I’m able to safely remove and re-add bigger servers in the cluster, refreshing the underlying physical machines while still keeping the safe 3x node quorum with a short 1-month period of ownership of new and old machines (while I move things around);

Yes, but cloud …

How often do you need auto-scaling? Never is the answer for my workload and some of the databases (H2 embedded) that power one of my side businesses are a few million entries in size. The sites, backed by MySQL or PostgreSQL are no that heavy either (only static articles)

Of course, you could make that “killer app” at some point in time. Looking back I may wish to have “auto-scaling”. But honestly, this “killer app” won’t happen overnight (instant success usually introduces growth pains so big that it makes things an instant failure).

You will see it happening. You will have a chance of scaling-up and when you do, the cost of ownership of bare-metal/dedicated servers on Hetzner plus a small virtualization layer I tend to believe is going to be less (a lot less) than the same resources in one of the big cloud providers (GCP/AWS).

Of course, it depends on what you want.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.