Getting an AI product from a first idea to paying customers is hard work, and a lot of it is not the kind of work you expect. At the recent Data Expo conference, Dennis Maas, Head of Product at Wolters Kluwer Schulinck, and Vincent Hoogsteder, Partner at Mozaik, shared what we learned from working together on exactly that, and gave a realistic account of what it took to get an AI product from concept to market.
Here is a summary of the main points:
The 10 Pitfalls to Avoid:
- Measuring quality at the water cooler: Subjective feedback is a shaky basis for decisions. Set up an evaluation framework early, so discussions about quality are replaced by automated, objective measurements.
- Prioritizing tech leaps over small tweaks: New technology is tempting. In practice, an experimentation mindset where even small tweaks to prompts and content count often gave the best results.
- Being complacent about the speed of learning: Waiting weeks for feedback slows everything down. We brought legal experts into the team so that feedback became daily and personal.
- Thinking from existing paradigms: A common mistake is to use AI to improve existing processes, like creating content summaries. The breakthrough came when we shifted the focus from the product to the customer's workflow.
- Choosing features over quality: New features are tempting too. The team went "all-in" on quality instead, with a "less is more" approach.
- Letting managers decide what to build: Managers rarely have the deep technical understanding these projects need. The people with the most knowledge, engineers, product managers, designers and legal experts, took the lead.
- Forgetting the human element: Fear of the unknown is a real hurdle. We designed the AI as a "CoPilot" and taught it to say "I don't know" and refer users to legal experts for sensitive cases.
- Testing and releasing like traditional software: AI is non-deterministic, so it needs a different approach. They set up analytics covering evaluations, latency, feedback and engagement, so they could act on any significant change.
- Overestimating the importance of latency: The first versions were slow, with response times of over a minute. We released anyway, despite our worries, and customers were overwhelmingly positive. Quality mattered more to them than speed.
- Believing too much of the LLM vendor marketing lingo: New and "better" models come out at a rapid pace, and they can have unexpected effects on quality and latency. We built infrastructure that makes switching between LLMs easy, and we test every new release thoroughly.
Ready to learn more?
For the full story behind these lessons, you can download the full presentation slides here.
