blue and brown concrete stairs

Continuous integration hits 30 years, though some still ignore it

It still puzzles me how is it possible that an 1991 idea, is still ignored by some software engineering teams worldwide in their advent to “deliver” at any cost, fail to invest the adequate time into ensuring they have all the pipelines in order. Don’t know if to attribute that to bad management or complacency inside the entire team.

Sadly to say, I’ve had my share of this situation. Though on the projects under my team’s ownership, we’ve been all the rage with tests, code quality metrics and documentation published directly from the source code, not the same I can say for the other half and for the artifacts produced in the last years of work, which are hand built, locally on the developer’s laptop which is releasing the artifact.

Continue reading →

Wildcard DNS in Let’s Encrypt with Go.CD, Ansible, FreeIPA and S3

When I started working on my own home-cloud (a weird term for a small self-sustained, bare-metal paid cloud on Hetzner) I needed a way to have trusted SSL certificates. I really, really hate the warning messages of the browsers when entering an self-signed site. One of my goals was to use Let’s Encrypt, put HAproxy in front of any and all services and have HAproxy do the SSL termination (and even internally, to have all services use Let’s Encrypt signed certificates).

As part of this small architecture (based on Proxmox in a cluster configuration) it was chosen also to deploy a 5-node FreeIPA cluster to manage DNS mostly but also I took advantage of other IdM features. Another goal was to implement the wildcard DNS challenge so that I wouldn’t have to configure each and every sub-domain I required (there were a couple of TLDs and a miriad of sub-domains which I already forgot their names).

Continue reading →

Continuous delivery of infrastructure as code using Go.CD and Ansible

I’m fond of the CI/CD movement, mostly because I can quickly see the value in automating the build and deployment pipeline and getting a quick feedback and if all tests pass, a good feeling of reliability of the service I’m deploying. A few years ago I would’ve used Go.CD for both CI and CD pipelines and I have yet to see a project that does not benefit from this ideology in some way or form.

The history of Go.CD starts as CruisteControl, probably the first CI software that was built in this industry, long before Jenkins became popular. Born in ThoughtWorks, backed by Folwer & friends, originally named Cruise in homage to the original CI tool, but quickly renamed to “Go” to avoid the confusion.

Continue reading →

Idempotent LXC with Ansible and Proxmox using “pvesh”

Back a few months when I started my Hetzner deployment of a small Proxmox cluster I checked to see if there was an Proxmox module for Ansible. And indeed there is one on the official documentation but as I was soon to discover, it didn’t work with my Proxmox 6 installation due to issue #59164 which got resolved (but is only available in 2.9.2 which my Debian-based Go.CD agents can’t see right now). Of course, I could install from “pip” sources and that would solve the versioning issue, but back then this was still an issue.

So what I wanted is an idempotent way of creating mostly LXC containers using Proxmox. Initially I wanted to go the REST API way but it was kind of complicated (in the sense of doing that from Ansible code). Secondly, there was the ‘pvesh’ CLI tool that we could use and based on the available “nextid” command I was able to “test” if the declared “vmid” existed:

Continue reading →

On workflow engines and where Airflow fits in

With the occasion of the CrunchConf 2018 there was a presentation on “Operating data pipeline using Airflow @ Slack” fromĀ Ananth Packkildurai. If you don’t know what Airflow is, it’s an workflow engine of the similar likes of Oozie and Azkaban. It’s based on the concept of a DAG which you write in Python and execute on a cluster.

As in the case of the Kafka presentation by Tim Berglund, we’ve asked the hard questions and they got popular pretty soon. In the case of Airflow, in the eco-system of workflow engines, we had quite a heavy question.

Continue reading →