By Filip Stachura, Appsilon Data Science.
This article presents our opinions and suggestions on how an Advanced Analytics department should operate. The post is not intended to be a comprehensive list of steps, but rather, a list of tips and warnings. We hope this will be useful for those who want to implement analytics work in their company, as well as for existing departments.
The post is divided into 3 parts. The first provides a list of the most important aspects to be aware of when leading AA work in your organization. If you are already leading such a department, then you are already aware of these but it can still prove useful when presenting your case higher in the hierarchy or to another department. The second part will include a list of the most important elements that need to be addressed and cared for in such a department. Lastly, we caution you of the most common issues we have seen in failed initiatives.
Part I - why should you care?
The most significant benefits are for business. One of the most important challenges is meeting management and C-level expectations. A proper implementation, does, however, increase the chances of success and can even guarantee returns in the initial phases. AI and advanced analytics solutions are very tempting but not every company is ready for them and they can be expensive. A company should determine their current level of progress. Based on this, they should determine what the logical next step should be. Trying to do too much at once carries increased risks. However advanced analytics is a good starting board for getting into AI.
New business models
Machine learning and deep learning, in particular, have allowed for completely new possibilities in the realm of predictions. In addition, companies are collecting more and more data. It’s estimated that the planet generates over 2.5 million terabytes of data every day (2.5 exabytes, References 1, 2, & 3). The rate of growth is accelerating (we double our data every 2 years - Moore’s Data Storage Law) at an exponential rate. Thanks to that it is possible to train analytical models that even a few years ago would have been impossible. Business has taken notice of their data sets’ power, leading them to develop completely new products and initiatives.
Reduce manual work time and errors
The truth is that most of the companies still heavily depends on manual work. We’ve seen tons of spreadsheets and understand that they are not going anywhere anytime soon. With that said, it’s hard to justify staying solely in spreadsheets in 2018. We’ve seen real-time dashboards replacing monthly spreadsheet summary. We are sure teams currently pasting data into cells every week and every month would be happy to automate these tasks and start analyzing data within. Otto, a German e-Commerce shows to what extent it is possible to minimize manual work and improve business metrics. Our intuition tells us that this would replace employees with algorithms, but no. They even increased their workforce.
Your competition is already working on it
The last and potentially strongest argument why this is worth considering: Your competition is already working on this. We’ve analyzed and worked with tens of different industries and identified the leaders within Advanced Analytics and AI. These companies are pulling away and experience similar compound returns as the aggregate data we generate. It’s hard to tell whether they will maintain their lead but we are certain that companies who refuse to take steps in this direction will not bode well.
Using data is proven to work. Our client’s examples showcase the possibilities. We’ve built and implemented a dynamic pricing model that deals with over 2 million quarterly pricing decisions. Increased fraud detection from 50% to over 90% and more accurately predicted e-commerce sales a year in advance. Our portfolio includes AI and AA projects for a large range of industries, often times including industry leaders.
If you discover more frauds than your competition – you get an advantage. Moreover, you build from there. You get new ideas every time you work with data. The power and value of using data are spreading within organizations as managers start noticing results.
How to get an advantage?
The very first thing is to understand where you stand today. It might be that you already gather a lot of data, but your organization is mostly driven by spreadsheets. This is how a vast number of organizations are managed. The good news, in this case, is that your next steps are easier to implement and the costs are significantly lower.
It may be that your company is already doing predictive analytics and you are thinking about prescriptive solutions. As such, you already have a large understanding of the benefits and amount of work needed to take it to the next level.
A Gartner (see Reference 4) breakdown of the four phases of data science projects may help you determine your current position. Treat this as an approximation as it may be the case that different departments are at different levels.
There are four fundamental ways of creating and using insights:
- descriptive - one that focuses on gathering facts about past. Most of the analytical dashboards work this way. They present read-only data with high-level metrics.
- diagnostic - this one makes a use of the data to understand the reasons behind the observed values.
- predictive - using forecasts is the first step to directly influence the decision process. Creating forecasts always require analytical model underneath. Quality of our decisions depends on the accuracy of the model.
- prescriptive - leads directly to a decision on suggestions or automation. In the first case, a decision support system is created that closely cooperate with a human.
Take note that earlier phases are more heavily reliant upon manual work and regular processing of the same tasks. It is the more advances steps the truly automate work. At these stages, people are crucial in defining problems and finalizing a final judgment.
Have a clear goal.
One of the biggest issues in AA departments is that their existence is a goal in and of its self. Such a department is most useful when its operations are close to the core of companies existing problems. It’s important to identify a few key areas where you expect analysts to be able to work in. Such a goal should also be attainable. You will have a difficult time achieving expected results. It will be difficult to achieve Prescriptive results if you are still in a descriptive phase. Such a project will also be negatively received by the rest of the organization because of workplace politics, a lack of understanding, or other reasons.
Understand the difference between development and research.
Communicate it clearly to others. One of the most difficult situations an Advanced Analytics department may find itself in is agreeing to work on a difficult research-intensive project in the same way as traditional software development projects. This can be increasingly difficult as a large portion of tasks does indeed have a software development-like nature.
Software development-like work:
- Interface elements, such as building a dashboard or API, UI/UX
- Infrastructural work, for example, the environment for reproducible research
Work of a research nature that should not be treated as engineering:
- building analytic models.
- data processing and validation