An open empty notebook on a white desk next to an iPhone and a MacBook

Orchestrating notebooks with Camel K, an integration-based approach

When you achieve somewhat of a level of maturity in your data analytics pipelines you also tend to start exploring various and flexible ways to orchestrate the ETL processes of data you have and derive various tables for different access patterns, as required by the business downstream. However, similar to the “object impedance mismatch” in the object vs. relation database worlds, there’s an “impedance mismatch” between data-engineers and business folks when it comes to the expectations they have on the speed of delivery, quality, correctness, maintainability.

In the mindset of the business folk, the derivative work or data that he requires is just a simple SQL that is run on the “raw data” which has infinite amounts of CPU power, and infinite amounts of memory, probably running on GPUs anyway and thus just needs to be written/queried as such and it will return results in an instant.

Continue reading →
closeup photography of brown analog scale

Scaleway Kapsule and Rancher-managed Hetzner Kubernetes clusters

I honestly think this is the 3rd time I’m moving things around. First I was on Proxmox managing myself a cluster of bare-metal Hetzner (from the server auction page). Then I was torn between a home server and one on Hetzner. I was using many images from the TurnKey Linux project. Nice, interesting but required immense investments of time.

Then I decided I wanted managed Kubernetes services. Not long ago (half-a-month) did I went over to Digital Ocean’s managed Kubernetes service. For not a small price, got a 3 node cluster with 24GB of RAM and 12 cores. Started installing my stuff and quickly ran out of resources only to be forced to pay more.

Continue reading →
close-up photo of gray stone pile

The dynamic equilibrium of code, flexibility in software development

It must be considered that there is nothing more difficult to carry out nor more doubtful of success nor more dangerous to handle than to initiate a new order of things; for the reformer has enemies in all those who profit by the old order, and only lukewarm defenders in all those who would profit by the new order; this lukewarmness arising partly from the incredulity of mankind who does not truly believe in anything new until they actually have experience of it.

Niccolò Machiavelli (1469 – 1572)

The above words are as true for software development as they are true for politics and human nature. Though unlike politics which requires a shift of mentality of the majority of the population and unlike the human nature which usually takes years to change or a very aggressive and disturbing moment of life (death, pain, illness) … we are “blessed” let’s say, in software development to have a choice.

Continue reading →
white ceramic mug near tea pot on table

It’s not just about learning a programming language

It’s actually teaching yourself a way to think … I found myself today in a situation at work that’s outlined perfectly in this article on strategic domain driven design. In fact, the original idea I remembered it from Eric Evan’s great DDD book, but I couldn’t quickly find the page and paragraphs that contained the idea of contexts and upstream/downstream systems so I relied on Google to find me a relevant authority to invoke. Luckily, the above article came about and it captured the idea I wanted to express to my co-workers perfectly.

The situation was that an upstream data producer made a deployment, to which we scaled effectively and nicely all in an automated fashion but then a day later, they rolled-back the change as it produced 50% extra data. Then at the roll-back, 50% of the data was gone.

Continue reading →