One of the challenges of shipping AI-enabled experiences is the shift in product management practices. As a fairly new domain, most product teams are not well-equipped to handle the end-to-end delivery of an AI/ML project. This leads to a lot of frustrations, quality issues, and worse, declining stakeholder's buy-ins in the organization. Hence, the followings share some of the key tips distilled from my own experiences building and shipping AI/ML projects.
Assess if it’s worth investing AI/ML
One of the biggest mistakes is overestimating the potential business return an AI/ML project can bring without prior validations. While there is no single rule to approach this, most projects fall into the following buckets:
drive business revenue (e.g. e-commerce product recommendations)
move key metrics to drive growth (e.g. LinkedIn's people recommendations driving virality);
increase the stickiness of product (e.g. Gmail's Smart Compose, LinkedIn's Smart Reply)
build a moat for the company (e.g. Gmail's Smart features);
strategic alignment
To ensure it's a problem worth investing months of engineering effort, it's important to consider the following factors:
expected return to the business (e.g. can you tie this to business-level metrics)
problem statements (e.g. are you solving a pain point or offering a delighter)
the opportunity cost of pursuing this (e.g what other user or customer problem you’re giving up by committing your time to this)
engineering costs (e.g. how long this will take, what are some of the technical challenges, GPU cost)
risks (e.g. is this feasible, data privacy, legal compliance)
These will help you think about whether it’s something worth investing months of effort into. As AI/ML projects have longer timelines than other software projects, make sure you do this with rigor so that your team gets to work on what matters most to the business.
Adjust expectations accordingly
AI-enabled experiences tend to fall prey to misalignment over what the product promises and what the experience is like. This can generally be solved with thoughtful design. While product managers don’t need to be well-versed with the design details, it’s worthwhile to know some of the best practices to ensure a smooth collaboration with your designers. This is especially handy when your designers don’t have any prior AI/ML experience. As a rule of thumb, you should be mindful of these two factors when discussing product designs:
metrics of the model that'll go into production, such as precision and recall;
user context and goal when carrying out a particular action
, so that you can close the gap between what the user expects and what the AI/ML model can offer. When in doubt, here are some of the design guidelines I find particularly helpful:
Update periodically
As illustrated in the image below, "you are what you eat" equally applies to your AI/ML project. Leaving your AI/ML model as it is post-launch without frequently revisit and update it risks jeopardizing your product quality.
Similar to the image above, you might notice that your prediction is suddenly suggesting responses such as “I’m a unicorn” just because there’s a running joke happening in the product recently. Yikes. Don’t forget to revisit your model just because it’s launched and marked as done in your project tracker.
Iterate with in-product feedback
One of the essential components to ensure your AI/ML model can continuously and effectively improve as you scale is providing in-product feedback. While this varies based on the type of product, the image above shows one of the typical examples of in-product feedback to improve feed recommendations. This allows users to feel a sense of control over what is suggested, whilst allowing engineers to improve the model quickly.