Data engineers and their unlocking potential for business use-cases

IMG_20181030_141321Nate Kupp currently holds the position of Director of Infrastructure and Data Science at Thumbtack and has presented this year his talk and success story entitled: “From humble beginnings: building the data stack at Thumbtack”. This is one of the presentations I’ve enjoyed much because it was similar to one of the pains I’ve also experienced in my day-to-day work.

A difference between Nate’s approach and mine is the executive sponsors (and a bit of luck of being in the right place, right time and the right management mentality). My experience on the other hand is, from my perspective a failure, but for others a small success against overwhelming odds.

Continue reading →

On workflow engines and where Airflow fits in

With the occasion of the CrunchConf 2018 there was a presentation on “Operating data pipeline using Airflow @ Slack” from Ananth Packkildurai. If you don’t know what Airflow is, it’s an workflow engine of the similar likes of Oozie and Azkaban. It’s based on the concept of a DAG which you write in Python and execute on a cluster.

As in the case of the Kafka presentation by Tim Berglund, we’ve asked the hard questions and they got popular pretty soon. In the case of Airflow, in the eco-system of workflow engines, we had quite a heavy question.

Continue reading →