Wading through the hype to find the best solutions for your business

Ashley Kilgore:

Operations research, analytics, and data science can provide a wide range of solutions to many of the challenges facing organizations from staffing to resource allotment to evaluating success and failure. If you have a business problem, OR and analytics can provide an effective database solution. But making sure you’re picking the correct solution, the right approach for your unique problem, is just as important. Sure, having AI or machine learning based solutions might sound super cool, but how do you know it’s actually the right answer to your problem? And when organizations are investing thousands, if not more in these solutions, a lot is riding on the choice you make. We’ll be exploring the answers to these questions and more with a particular focus on data-centric versus decision-centric analytics and solutions with Karen O’Brien.

Ashley Kilgore:

Karen O’Brien is an analytics leader and 20 year public servant, having served in diverse analytics and leadership roles within the Department of the Army as a member of the Army Civilian Corps. Her applied operations research experience ranges from ballistics to logistics, and she has built advanced data analytics capabilities and multiple organizations. By training, her undergraduate work was in physics and chemistry, and she leverages her masters in science and predictive analytics from Northwestern to bridge the worlds of big data analytics and classic operations research. Karen, welcome. It’s so nice to speak with you. I have a feeling I’m going to learn a lot from you today.

Karen O’Brien:

Thank you for having me. This is a wonderful opportunity.

Ashley Kilgore:

Let’s start with an explanation of data-centric versus decision-centric analytics.

Karen O’Brien:

Sure. This is a conceptual model that is found in the 2019 INFORMS Analytics Body of Knowledge, and it’s specifically chapter one that was written by Keenan, Owen, and Schumacher. And this is the most amazing little model. I remember when I first got a copy of this book, I was riveted. I could not get past chapter one, because that conceptual model was so useful to me in breaking through some barriers to communication. We were working on a data analytics problem, and we had one group of people, and they had one concept of what data and analytics was and a different group of people who had a different concept. And we were using the same words but meaning very different things. So everyone was at an impasse until we drew out this model on the board and created a common mental model of data and analytics. This model functions in many ways like those Gestalt shift drawings, where you look at it and depending on where you put your focus, it looks like a duck or it looks like a rabbit.

Karen O’Brien:

It’s the same drawing, but your attention shift reveals a different aspect of the drawing. So this model helps you shift your focus so you can understand the bigger picture of data and analytics. I’ll describe this a little bit, so either grab your copy of the Analytics Body of Knowledge, turn to chapter one, or imagine if you will, a circle. The circle is divided into quadrants. At 12 o’clock, you have one node that’s entitled Decision. Proceeding around to the right clockwise to three o’clock, we have Data. Down at the bottom at six o’clock, we have Analysis. And at nine o’clock to the left, we have Insight. For the purpose of this model, clockwise is going forward and counterclockwise is going backwards. And the way you use this model is you start in different places. So the data-centric analytics paradigm starts on the data node, and then you proceed clockwise from data. With data, you can do analysis and from analysis proceeding to the left, you get insight.

Karen O’Brien:

And with those insights, you can make decisions. Conversely, the decision-centric analytics approach runs that process backwards. This time we start with decision and work back to insights and ask, “What insights do we need for this decision?” Once we know what insights we need, we can figure out what analytics methods are helpful in producing those insights. Then going back even further, then we can identify the data we need to do that analysis to produce the insights and make the decision. So spoiler alert, by the time you get to the end of the chapter, we all have to conclude that you need to use both of these techniques and use them both together. Because each technique, each approach has pros and it has cons, and the strengths and limitations of each paradigm compensate for the limitations of the other. So the strengths of one compensates for the limitations of the other.

Karen O’Brien:

In a nutshell, data-centric analytics starts with the data. And this method is well-suited for leveraging large amounts of data. In fact, more data is better. And in a data-centric analytic organization or an ecosystem, you see databases that are organized, data architectures, data engineers. You have data extraction, you have the whole ETL pipeline, you have models, you have dashboards. It’s a huge ecosystem with wonderful technology and experts. And another strength of this is that it’s fast. It lends itself very well to automation. So you can set up persistent visualizations, persistent calculations, and build a lot of automation around things you know you’ll need. As a result, this type of approach is suitable for things like artificial intelligence and machine learning, which has the potential to extract really weak signals from noisy data, and you need a lot of data in order to do that. Strengths of the decision-centric approach where you’re starting with the decision and working back to data, it allows you to leverage domain knowledge and subject matter expertise.

Karen O’Brien:

Big data isn’t great here. Curated data is better. Smaller amounts of focused data works better. In addition, the decision-centric approach is context aware. It knows what’s going on in the world, because you have humans thinking about the problem. Your approach to solving this problem is aware that maybe there are supply chain disruptions, maybe there’s a global pandemic, maybe there’s inflation. So you can build some of these insights into your analytic approach. You can’t really do that when you have an automated system generating forecasts or calculating metrics for you. This is also a helpful approach, a decision-centric approach, when you don’t have precedent, when it’s a new question, novel question, something that’s never happened before. It allows you to work through that unique case, and it gives you those strategic insights. And ultimately, there’s no limit to the questions that can be asked. Pretty much anything under the sun you can approach with a decision-centric approach.

Ashley Kilgore:

So now Karen, could you share what might be some of the limitations of these approaches?

Karen O’Brien:

A data-centric approach ultimately has a finite list of questions that you can ask and answer, because you don’t have infinite amount of data. The data you have reflects decisions about what data to collect years ago. So maybe you’re working with a database that was stood up 10 years ago, 15 years ago, 20 years ago. And data storage was much more expensive back then, so you had to be very choosy about what you did store. And that may constrain your ability to ask questions. You might not have the data you need to answer them. Conversely, over in the decision-centric analytics, the big drawback of that is that it’s slow. If you have to pull subject matter experts together, maybe bring in consultants to work through how to solve the problem, you’re not going to be able to cycle through to answers as quickly as the big automated data systems.

Karen O’Brien:

You may find that the data you need doesn’t exist yet. It doesn’t exist in the databases, and you might have to generate data. You might have to have a data generation event. So you might have to do a survey. You might have to do a focus group. You might have to do research, have experts come in and make observations. And it’s much harder to automate this type of thing. Now, by using both of these together, you can leverage the fast analytics that the data-centric approach gives you, but then you can also increase the data and options that you have by spinning off requirements from that decision-centric approach. So by using both of these approaches together, you end up with being able to do more analytics faster over time.

Ashley Kilgore:

So now in a business or say public sector organizational setting, how are these different approaches used? What kind of solutions can they provide?

Karen O’Brien:

You tend to see, or it’s been my experience, that an organization will gravitate towards one or the other, depending on that organization’s mission in the world or their history. And you can really spot the assumptions and their preferences in the language that they use. If you’re talking about extracting value from the data or digital transformation, it might be a data-centric organization. They will likely have data organized in the information systems and it’s piped into models and the results are visualized or pulled into a product to make it better. Then typically, they use data to support recurring or persistent information needs. They might focus on metrics or trends or forecasting, and visualizing the data in a dashboard is common. And that tends to support the subset of decisions that benefit from descriptive analytics. Other ways to use that data in a data-centric culture is piping it into a product, improving the functionality of it.

Karen O’Brien:

It might be the product itself, or it might make a product better. And a good example are the product recommendations that you’d find on your favorite shopping website. That’s a result of a data-centric capability. So it’s really important to note that since the data-centric solutions tend to be the focus and the dominant voice, there is a huge marketplace. It’s just brimming with fast, easy to implement, proven solutions that you can get up and running. Data analytics platforms, low code support, and these are profitable for vendors, and it can result in a real significant increase in capabilities for organizations. That’s why we gravitate towards them. And over time, this approach can be a rising tide that lifts a lot of boats. On the other side of the house, the decision-centric paradigm often gets characterized as consulting more than big data analytics. An analytics consultant can help you work through framing the problem statement or bringing structure to an unstructured problem.

Karen O’Brien:

The data may not be present in the database, so someone needs to help do an analysis to figure out what data you will need, what methods you will need, and how to get all that up and running. So that’s commonly thought of as consulting rather than analytics, but it is a way to do analytics, especially for things like novel questions and things the organization hasn’t faced before. With this approach, you tend to have a much bigger menu of analytics methods that are available. You’re not limiting yourself to a statistical forecasting method or AI/ML. You have the full range of qualitative and quantitative and mixed methods. So maybe the solution to this novel question is a survey, or maybe it’s an AB test, or maybe it’s a robust machine learning time series forecast that’s tied to an optimization model. Really, there is no limit. And the goal is to support the decision, and it may generate those new requirements for the information systems to implement and new metrics to visualize and so forth.

Ashley Kilgore:

Now, as we talk about the business and public sectors, I know AI and machine learning are really becoming buzzwords, even outside the world of analytics and OR just like the term supply chain has following the pandemic. But while more and more people are familiar with these terms, especially in a business setting, that doesn’t necessarily mean they understand what they are or how they can help their business. What do you think are some of the biggest mistakes businesses can make when choosing a data science based solution to a problem?

Karen O’Brien:

Well in my experience, the biggest mistake is not pausing to reflect and evaluate your needs before investing in a solution. Whether it’s a capability of vendors going to provide to you, or whether it’s something you develop in house, it’s worth doing that. So that is that decision, that deliberative decision-centric approach, where you figure out what you want to know, and then figure out how you can know that. Then, you can go get your data and build your pipelines. To do this, you might need to hire a consultant, because that is often thought of as consulting. But this is something an operations research expert, particularly a certified analytics professional, can help facilitate getting this process going to frame the problem and then recommend the analytics methods that can meet the need. I’ve seen money spent in organizations on a dazzling, beautiful dashboard and all of the supporting infrastructure for it.

Karen O’Brien:

Then, ultimately come away disappointed that it’s not actually helping them make decisions. So to invest your money well in your time and your resources, figure out what you really want to know. The decisions you want to use those data analytics solutions for before you go spend a dime. Additionally, AI/ML is the perfect solution for some problems. It’s the absolute worst choice for other problems. And for a lot of the problems in between those two extremes, you can use AI/ML in combination with other tools, like OR tools. Sometimes what you need is an optimization or a simulation and an OR expert or a certified analytics professional can help you figure that out. AI/ML might not be that complete solution, but it might make another model even better. Like if you have an optimization model that you populate with AI predictions, and there’s a lot of good examples in industry.

Karen O’Brien:

My favorite is DoorDash. They have a central optimization model. It’s an assignment problem, but they populate that model with predictions that they get from AI. Models like that are worth looking into, because they pull the best of breed in order to solve the problem. They don’t presuppose that one method is going to get the job done.

Ashley Kilgore:

Now, as we’re discussing data-centric versus decision-centric analytics in the context of decisions, there’s obviously often a great deal of money associated with implementing new solutions. What are some of the challenges and complexities or even questions raised if the person making these decisions is maybe not well-versed in analytics?

Karen O’Brien:

Well, in the commercial sector, you’re accountable. You’re accountable to shareholders, you’re accountable to your customers, and in the public sector, which is where I work, we are public servants, and we have to be accountable to the taxpayer. We have to be stewards of the taxpayer’s resources. And being a good steward really demands we have enough training and analytics that we can make good decisions about what to buy, that we can select the right tool for the job, and invest the money wisely. This is something I feel very strongly about, so an analytics education even at the executive level can reduce risk and help you spend money wisely and have a better chance of getting a good outcome from your efforts. This does require us to get deeper than just the pitch deck for an off-the-shelf, AI/ML pipeline. Those technologies are mature and robust and easy to implement, but you need to know what analytics techniques are useful in answering questions that start with the word, “Why?” Or “How can I make the best choice?” Or “How can I manage risk?” Or “What problems are coming at us that we should get ahead of?”

Karen O’Brien:

We have to make decisions today in order to preserve our decision space for tomorrow. Analytics professionals know there are operations research, management science methods for some of these. An optimization model, a simulation. We also know that there are some fantastic AI/ML methods, and when you combine these different methods, stack them together, you can end up with very powerful solutions. And again, a consultant or certified analytics professional can help with this. So currently all across the federal sector, there is pressure to “do something” with data and artificial intelligence. And there’s a rush on to find good use cases. Sometimes the use cases are good, sometimes they’re bad, but just to show that we’re doing something with AI. In a way, this has created a third way to use that data-centric versus decision-centric model. So in this third paradigm, you start on the node called Analysis and then cast out in both directions.

Karen O’Brien:

You have to hope that your use case… And you’re presupposing a method, artificial intelligence, for instance. So you have to hope that the data is there, that the data’s fit for purpose to support this project. And then, you have to cast out in the other direction and hope that it will actually connect with a decision that needs to be supported. When you’re doing this, there’s a high risk of failure, of falling short. The data’s not there, or it really doesn’t meet a need at the decision level. And a couple years ago, 2018 I think, Gartner provided an estimate that 85% of data science projects will fail. Well, I think a driver of this trend is presupposing a data science solution or an artificial intelligence solution without first doing that problem analysis. Starting with the decision and working back to the data.

Karen O’Brien:

So if the need is for a causal analysis, but the mandate to “do something” with AI drives you to an unsupervised machine learning thing that can’t explain causes. Then yeah, you have failed to show value. You’ve failed to extract that value. So my recommendation is to start with data or start with decision, but don’t necessarily start with a technique. Assuming or presupposing that AI/ML is the thing you need can create waste.

Ashley Kilgore:

Karen, thank you again so much for joining me. This has been super informative, and I really did learn a great deal during our conversation today so thank you, especially for that. Now, to wrap things up, what guidance or insight would you share with organizations that are looking to OR and analytics to help address their challenges?

Karen O’Brien:

Well, this has been fun, and I appreciate the opportunity to share some lessons learned in this. It’s such an exciting time to be in the analytics business. We have never had such a confluence of techniques, classic and advanced, and computational capability. It’s a great time to be an operations research professional. In addition, the operations research and management science literature is deep. Not just with algorithmic approaches, but with conceptual models and techniques that can help with framing problems and working through what the right methods are to use. So the model we talked about today from INFORMS, Analytics Body of Knowledge, is one of the more recent ones, but there are many from the literature, including Russell Ackoff’s Data to Wisdom Hierarchy from about 1989.

Karen O’Brien:

That one particularly is helpful with aligning techniques to the decisions and questions that are getting asked. I think I’d like to close with an analogy from photography, which is to keep the aperture wide open. Grow your knowledge as much as you can, be expansive in your thinking, and have many analytical tools and models available. In photography, having a wide aperture brings the subject into focus while at the same time, blurring the background. And helping to focus on the actual business need generating actual business value is something that the operations research community has a proud history of doing and something that we’re well-placed to do now.

Ashley Kilgore:

Want to learn more? Visit ResoundinglyHuman.com for additional information on this week’s episode and guest. The podcast is also available for download or streaming from Apple podcasts, Google Play, Stitcher, and Spotify. Wherever you listen, if you enjoy Resoundingly Human, please be sure to leave a review to help spread the word about the podcast. Until next time, I’m Ashley Kilgore, and this is Resoundingly Human.