Will my AI project deliver its intended value?
Step-by-step guide to tackle AI product risks
Overview
A whopping 87% of AI projects are expected to fail, leaving most product teams to apply AI technology that yields subpar returns to organizations. In addition, knowledge sharing on product lessons in this space is not prevalent enough. With so few resources out there, how do you know if an AI project is worth pursuing?
When I first asked this question, I knew very little about how to evaluate when to build what in the context of AI/ML. Thankfully, after working on numerous AI/ML projects and making my fair share of mistakes, I now take a more deliberate approach that I am sharing here.
Step 1: Evaluate current state of product
Pursuing a project that doesn't fit the current lifecycle of the product, especially in the context of AI/ML is extremely costly. While this is the premise of the KANO model, it's worth emphasizing that poor judgment of when to build what can lead to low adoption, thus making the iteration of the AI/ML model difficult.
The diagram above maps the product lifecycle to key factors to consider. In the introduction phase, there's generally not much real AI/ML application to do. To deliver business impact, consider leverage external data or mature AI/ML technology. For example,
In the growth phase, AI/ML has great potential to accelerate product growth and build a hard-to-replicate technology moat. For example,
LinkedIn’s people recommendations tapped into virality to accelerate growth in the early days of LinkedIn
Meituan’s DianPing recommendations built a strong moat that's hard to replicate by competitors, attracting merchants to its platform that heavily rely on the platform to conduct business
Dovetail’s automated transcripts use automatic speech recognition to transcribe interview notes to significantly increase the number of research content on the platform
In the maturity phase, AI/ML can help unlock a new frontier of growth. For example,
Gmail’s smart compose created a new wave of growth for Gmail by driving up users engagement
Step 2: How would the world be different?
Upon understanding and mapping the end-to-end workflow of your target users, try asking:
“How would the world be different if we can apply AI technology to the current context?”
This will help unlock immense value when thinking about potential solutions as a team (e.g. the 10x mental model) and is especially important in the context of AI/ML projects. To illustrate, here are some examples:
Unlocking a phone with a passcode wasn't a big pain point, but FaceID enabled us to unlock our phone quickly & securely
Writing reasonably well required us to get better at spelling and grammar, but we can now delegate such task to Grammarly to focus on our creativity and content
These didn't seem like noteworthy pain points that users are screaming to us at, because most people got used to how things are supposed to work. It is for this reason that product teams who can shift the way of solutioning would be able to unlock greater AI/ML potentials.
Step 3: Understand current state of technologies
It’s important to recognize and separate between easy, hard and impossible technical problem that may hinder you from the desired user or business outcome. For example, in the context of natural language processing (NLP):
Providing response suggestions based on previous chat history of a conversation is easy;
Providing a predictive text based on a prefix of a sentence is hard;
Summarizing a long document to offer an accurate “too long, don’t read” section to your readers is currently an extremely challenging technical problem, hence almost impossible to deliver a decent experience to your users.
Overestimating what the technology can do leads to the danger of overpromising and underdelivering.
Step 4: Do we have access to the data required?
Do your research to identify if there are any external or internal constraints from data access. In particular, make sure you go through the followings:
source of data: in-product, third party integrations, and open-source
rules and regulations: data privacy, local laws (e.g. GDPR) that may prevent you from data access
stakeholders concern: any risks about mishandling or miscommunication of user data should be called out
All the above considerations should be tackled upfront and communicated to your team to enable them to do their best work.
Step 5: Cost-benefit analysis
Pro-tip: Do this as a cross-functional team to collect divergent inputs that you may have missed
Supposed that you’ve identified a project worth pursuing in the current stage of the product, how does the value and cost equation look like?
Business level metrics: can you tie this to the strategic business goal?
User level metrics: what’s the user level metrics that’ll prove that your project is successful? For example, can you measure the efficiency gained, user satisfactions or engagement? how much of an estimated improvement can we make?
Cost of data processing: while often neglected, it should be included as part of the cost estimation, such as headcounts, GPU costs, data warehouse, purchase price of external data
Engineering effort: this is usually self-explanatory, and can be measured by headcounts and estimated time required
Conclusion
I hope this guide serves you well, so you can avoid preventable mistakes throughout your product journey. If you are working in an AI/ML product team, I’d love to hear what you've been doing that's working well on your end. Do leave a comment or drop me an email at pies@substack.com to submit topics that you’d like me to cover here.