AI Development Process
FAQ on how to capture AI benefits for business
AI Development Process – Frequently Asked Questions
If you work with an AI consultant, you may have questions. Below we outline our method and how we collaborate with your team.
How do you plan an AI project?
We define the business problem we are trying to solve with AI. The modeling target is what we want to predict. We plan for a solution that is consistent with the way managers think about the industry and the value the company provides. This is how we arrive at an approach that delivers AI benefits for business.
KEY AI TASK
Discovery: Interview executives to set vision and goals
- Define machine learning experiment at a high level
- Define project timeframe
- Understand the company’s capacity to implement a solution
- Ensure modeling process fits into the business
- Ensure intended AI solution is implementable within current business parameters
KEY AI TASK
Discovery: Interview business leaders about situation, causes, planned uses
- Refine definition of the modeling target
- Generate causal hypotheses for good and bad outcomes
- Start with strong signals that are understood by the business
- Educate the business and the data team on how AI works
- Leverage in-house expertise to get a quick start
- Tie in-house experts to the solution
- Get buy-in on the process and the outcomes
- Ensure alignment of AI with business so the solution falls within expectations
What does an AI consultant do?
As AI consultants, we use our experience in machine learning to develop and deploy predictive models. Our background in strategy consulting helps ensure the models address business challenges.
KEY AI TASK
Work in sprints to make progress
- Execute against clear milestones to advance the project through all stages of the AI development process
- Streamlined project management and alignment on what is important for project success
KEY AI TASK
Partner with in-house data experts
- Conform with known business and data join rules
- Conform with known limits on data reliability or meaning
- Get data faster, clearer, and in a way that is repeatable when production starts
KEY AI TASK
Prepare data and refine feature development
- Ensure data is prepared and staged to represent how the business actually operates
- Accelerate team process, try new things faster
- Derive new fields, join tables, feature optimization
- Let business users get back to their regularly scheduled work
- Separate prescribed work from experimental work
- Come back to the business with more specific requests as needed
KEY AI TASK
Weekly check-ins with working team
- Keep AI development process on track
- Get deeper understanding of data
- Control process and understanding around unexpected data patterns
- Fix data anomalies and improve data quality
- Keep the project on track
- Receive analysis and insights for managing the business better
KEY AI TASK
Add new data as it becomes available
- Gain richer description of reality to help models continue to learn
- Increase ability to detect “actionable pockets”
- Better business outcomes based on more accurate models
- More efficient field testing
What does AI model selection mean?
To create a final AI model, we evaluate up to a dozen algorithms. We may combine algorithms into ensembles, and apply different algorithms to subsets of the data. By the end of the AI development process, we may create hundreds of candidate models.
KEY AI TASK
Model selection: match input data structure to the modeling goals
- Develop valid models that align with the business
- Match target type to input data
- Obtain relevant insights and predictions in order to take action
KEY AI TASK
Iterate on modeling, data additions, model quality review
- Ensure ongoing model improvement
- Add new features as they become available
- Gauge early and incremental progress
- Simulate towards production readiness
- Test small use cases first to reduce full deployment risk
- Improve action and monitoring processes
- Improve outcomes and track effectiveness
How do you manage data for AI modeling?
We ingest data into a virtual private cloud, derive new features, stage it for machine learning algorithms, and document its use in the AI development process.
KEY AI TASK
Sample design for data pull & data processing
- Provide clear signals for the algorithms to learn from
- Potentially reduce size of the data while still keeping it representative
- Capture data that reflects true, real-time conditions of the problem and the target window, so the final output will work in real time
- Improve data processing time to save resources and provide time for iterations to improve model quality
KEY AI TASK
Sample design for modeling, training, validation, holdout, and field tests
- Ensure field accuracy of model can be assessed
- Find data preparation problems while model is being built to avoid failure in the field
- Avoid rolling out dud models
KEY AI TASK
Data visualization
- Visualize insights in graphs, workflows, diagrams to detect issues with data quality
- Discover new business insights
- Ensure data quality, improve data preparation
- Improve the business process through analysis insights, give more accurate models
KEY AI TASK
Develop or enhance data dictionary
- Maintain common understanding of features and their derivation
- Identify and correct misunderstandings
- Knowledge transfer back to data teams
- Makes it easier to deploy models in production
KEY AI TASK
Document AI process
- Make data processing consistent, repeatable, and faster
- Knowledge transfer from AI consultant back to internal teams
KEY AI TASK
Develop data notebook, experimenter’s lab book
- Check data assumptions, improve data understanding
- Raise questions for discussion and presentation
- Ensure data quality
KEY AI TASK
Use ML sandbox and secure data platform
- Maintain modeling datasets in a machine learning sandbox environment to provide flexibility to data scientists
- User identity management, multifactor authentication, and rigorous controls to protect data both in transit and at rest
How do you deploy an AI model?
To deploy an AI model, you have to evaluate model metrics and select the winning model. This involves tradeoffs in accuracy, relevancy, ease of implementation, and other factors.
KEY AI TASK
Model evaluation and metrics selection
- Reveal which models perform best with this data
- Define performance with different criteria including early lift, total lift at 40% of file, precision, recall, or f-test
- Use validation data or holdout-forward data to test the models
- Model competition gives best choice for accuracy and usefulness
- Holdout testing gives a clearer expectation on field performance and model profitability
KEY AI TASK
Stakeholder check-ins
- Ensure assumptions built into the data prep match meaning to the stakeholder
- Confirm new insights are not incorrect data prep
- Adjust modeling process, if needed based on stakeholder feedback
- Provide a sprint goal for the team
- Adjust expectations and track progress
- Share any business analysis that comes up to confirm that data prep is following conventional wisdom or revealing new issues
KEY AI TASK
Field testing
- Check assumptions in data prep, and in preparing the target
- Make sure model is set up to deliver insights the business can use
- Make adjustments to inputs or target, if processes are misaligned
- Use incremental change to adjust business practices gradually
- Make improvements to models and business process
- Share field test feedback to analysis and modeling teams, for continuous improvement of business results
How can I get help with an AI project for my business?
Working with an AI consultant can help you clarify your business objectives and confirm that machine learning and AI can solve your business challenges.
End-to-end benefits of our AI development process
Johnson Controls is using AI to reduce churn and identify over $100M a year of protectable revenue
