No skill left behind

Jeff Bezos is famous for Amazon’s focus on things that aren’t going to change in the next 10 years.

“I very frequently get the question: ‘What’s going to change in the next 10 years?’ And that is a very interesting question; it’s a very common one. I almost never get the question: ‘What’s not going to change in the next 10 years?’

And I submit to you that that second question is actually the more important of the two — because you can build a business strategy around the things that are stable in time … In our retail business, we know that customers want low prices, and I know that’s going to be true 10 years from now. They want fast delivery; they want vast selection.

It’s impossible to imagine a future 10 years from now where a customer comes up and says, ‘Jeff I love Amazon; I just wish the prices were a little higher,’ [or] ‘I love Amazon; I just wish you’d deliver a little more slowly.’ Impossible. […] When you have something that you know is true, even over the long term, you can afford to put a lot of energy into it.”

From the article: https://www.ideatovalue.com/lead/nickskillicorn/2021/02/jeff-bezos-rule-what-will-not-change/

Original video can be found here: https://youtu.be/O4MtQGRIIuA?t=267

I have been analyzing data since 2004 – basically all my professional life. I have seen the tools people use change (from Spreadsheets and calculators to reporting tools and infographics, to even more fancy BI tools and slick dashboards. At the same time, the methods we use to analyze data have changed from basic statistics (total, average, min, and max), to advanced statistics (distributions and variance), to advanced statistics, mathematics, and computer science (forecasting, predictions, and anomaly/pattern detection).

While the methods and tools to process and present that data have changed significantly over the years, there is one core skill that is increasingly becoming more important in terms of our ability to make meaning out of data.

That skill is the ability to do analysis, and the role is of a business or quantitative analyst. I know it sounds basic – almost too simple to write about. And therein lies the key reason why many enterprises fail to use data properly within their organizations. Yes, I admit that there are other roadblocks in a firm’s ability to use data – such as access to data, understanding the meaning and acceptable use of data, having the tools and methods to process the data, and finally, the ability (and time) to interpret the meaning of data. I contest that many of these challenges are symptoms of a bigger problem – the lack of a full-time role to analyze and make meaning out of data.

As we shifted to “Big Data” around 2010s, companies quickly pivoted to creating new roles to follow the trends. Hiring data scientists was a top-5 priority for almost any organization. Then came the realization that good data science requires data that is both broad in its sources, and deep in its volume. This followed the move to data lakes to bring all the data into one place. Many companies invested in large Hadoop-based infrastructures, only to find that once data is in one place, it’s almost too much to make sense of, and very expensive to maintain. Cataloging, searching, and distributing data proved to be a big IT overhead. With the advent of the cloud, suddenly there was a new way to solve the problem – migrate to the cloud, uncouple storage and processing, and save a ton on your overheads. The move to the cloud has proven moderately challenging for many customers, and frustratingly complicated for others. The skills to build, maintain, and evolve complex architecture is a full-time job that modern technology organizations should focus on 100%.

What has not changed during this time? It is still a person’s ability to ask business questions, identify the data needed to answer those questions, apply the stat/math/cs methods to process the data, and present the information in a consumable manner. That’s the job of a data analyst, and I argue we need more of them – that is a skill that does not scale non-linearly. At least, not yet.