Data Science 101 – Part 2: How can you build trust in data science models?

Many companies have come to see the value in the latest capabilities of AI and machine learning. However, making critical business decisions based on computer-generated predictions can be a tough sell across an organization. And yet, insights using data science models often drive significantly better results than best guesses.

The point of introducing AI into your business is to achieve better outcomes by improving the way you operate and make decisions. But it means putting new best practices into play that help people feel comfortable with the idea that machine learning models can provide useful insights. Machine learning is an iterative process that requires an internal partnership between data scientists, subject matter experts, and business managers.

So how can you build reassurance into your data science practices to help everyone see the business value?

Start simpler to build confidence

To help ease companies into a deeper understanding of what AI can provide, our data science team tries to avoid black box solutions for the first deployment of a model. They are highly complex and don’t allow you to see how the model determined predictions. Instead, we prefer to start with algorithms that provide explanations for their predictions. That way we can better validate the reasoning to ensure the model derived answers in sensible ways.

Once the model is consistently performing in line with identified metrics, and the data pipeline is stable and understood, you have the option of exploring powerful black box solutions such as neural networks. (As we noted in Part 1 of this blog series, you’ll typically want to use ‘easier’ algorithms to pilot and refine your initial machine learning model. Then once that model is in production and you’re ready to tinker with it more, you might try a black box algorithm.)

Grow trust in the model with the right data

Your data science models are only as good as your data. To get actionable predictions, your dataset for a specific hypothesis needs to align with real business practices and decisions. For example, if a coupon was offered to encourage a transaction, then the presence of a coupon and the amount of the discount should both be reflected in the data.

This is part of a “go slow to go fast” data science best practice that builds up trust in the data and the models over time. You can help grow trust in the data and models by examining them in a few ways:

• Business experts in your organization can assess if the predictions are sensible and actionable. That helps confirm the data has been prepared in such a way that the pathway to the prediction makes sense.
• Data scientists can check performance metrics of the model using split file and forward testing. Split file testing allows them to run statistical analyses on subsets of data. The forward method lets them test a predictive model by adding variables one at a time to evaluate, step-by-step, which ones produce the best improvements.