An open empty notebook on a white desk next to an iPhone and a MacBook

Orchestrating notebooks with Camel K, an integration-based approach

When you achieve somewhat of a level of maturity in your data analytics pipelines you also tend to start exploring various and flexible ways to orchestrate the ETL processes of data you have and derive various tables for different access patterns, as required by the business downstream. However, similar to the “object impedance mismatch” in the object vs. relation database worlds, there’s an “impedance mismatch” between data-engineers and business folks when it comes to the expectations they have on the speed of delivery, quality, correctness, maintainability.

In the mindset of the business folk, the derivative work or data that he requires is just a simple SQL that is run on the “raw data” which has infinite amounts of CPU power, and infinite amounts of memory, probably running on GPUs anyway and thus just needs to be written/queried as such and it will return results in an instant.

Continue reading →

Wildcard DNS in Let’s Encrypt with Go.CD, Ansible, FreeIPA and S3

When I started working on my own home-cloud (a weird term for a small self-sustained, bare-metal paid cloud on Hetzner) I needed a way to have trusted SSL certificates. I really, really hate the warning messages of the browsers when entering an self-signed site. One of my goals was to use Let’s Encrypt, put HAproxy in front of any and all services and have HAproxy do the SSL termination (and even internally, to have all services use Let’s Encrypt signed certificates).

As part of this small architecture (based on Proxmox in a cluster configuration) it was chosen also to deploy a 5-node FreeIPA cluster to manage DNS mostly but also I took advantage of other IdM features. Another goal was to implement the wildcard DNS challenge so that I wouldn’t have to configure each and every sub-domain I required (there were a couple of TLDs and a miriad of sub-domains which I already forgot their names).

Continue reading →