On data architects and their cost-saving role of fitting management requests in a puzzle of infrastructure and human resources

As stated earlier, Andrey Sharapov’s presentation on “Building data products: from zero to hero!” has given me many motives to talk. And I can’t seem to stop with the ideas of things to write about. Maybe probably because I’ve been, at my current employer, at the time of writing, through the same pains as the guys at Lidl did. And those pains are centered around the management of data people, be them engineers or scientists.

The featured image of this article present a young female manager asking: “How are you?” and getting a cryptic reply: “About half a standard deviation below the mean …” which in some languages goes by as swearing or rude behaviour. But in all honesty, although English, these people come from very different backgrounds, with slight variations of vocabulary (and understanding, not that one is less of an educated person from the other as management itself is a hard discipline also as it involves a few areas of sociology, economics and more).

Continue reading →

Unicorn data engineers & scientists, a guide to catch, keep and sh*t rainbows

This year at CrunchConf 2018 there was an interesting talk by Andrey Sharapov an Data Engineer & Scientist at Lidl. Yes, Lidl. The store in your back alley or in your neighbourhood. Did you know it does Big Data? I assumed, yes, given one wants to optimize both the idea of minimizing waste and increasing profits (eg. how much of X do one store needs to order to ensure it’s gone by EOD).

Andrey’s talk was centered around “Building data products: from zero to hero!” and I would personally want to apraise the realism of his presentation which gives me content for more than one article on the subject. He’s one in a series of presenters at this year’s conference that has called out to the strategy of companies of investing too much in data scientists, then finding out they don’t have an infrastructure those scientists need, then trying to find data engineers a bit too late in the game (which are even more scarce than scientists).

Continue reading →