Measuring What Matters: The Science of UX Design
“You get what you measure. Measure the wrong thing and you get the wrong behaviours.”
John H. Lingle
I start with this quote to establish the science of UX design, which boils down to one fundamental truth; removing as much ambiguity from the experience as the circumstances allow. This is what metrics can help with, when used properly. We’ve seen firsthand how critical it is to align metrics with the product design and development process. By doing so, we can ensure that our digital products and services not only meet user needs but also achieve business goals.
In this article, I’ll walk you through the key metrics to consider at each stage of product development and provide insights on which ones are “mission-critical” and which ones are “nice to have.”
Let’s take the example of a fictional company called “TrioLingo” (super original, I know) that’s developing a language learning app. Here’s how they might align metrics with their product development process.
Discovery Phase:
During this phase, TrioLingo should focus on user research metrics, such as the number of user interviews and surveys completed, to gather valuable insights into user needs, preferences, and pain points. These metrics help teams understand their target audience better and inform the product’s direction. The discovery phase will focus more on gathering research materials rather than metrics but making sure the beginning of the project is guided by a healthy number of user and customer research will help give TrioLingo a good chance.
Pitfalls to avoid:
- Market research at this stage isn’t a replacement for user research. And vice versa. It may overlap with some of the needs with market research and give you some insight on how TrioLingo will “fit” in the market BUT it cannot be used in addition. Our article “Drive Success and Avoid Costly Mistakes By Getting Your UX Research Right!” touched on the importance of getting your market fit right.
- The discovery phase is the foundation of every other step; it is how the idea takes shape. By investing time and resources into thorough user research and analysis, you can uncover valuable insights that will guide your product development process and help you create a solution that truly resonates with your target audience.
Design Phase:
As TriLingo moves into the design phase, wireframe and prototype metrics take centre stage. User task completion rates, time on task, and task-to-completion rates are mission-critical to validate design concepts and identify areas for improvement. Task-to-completion rates measure the percentage of users who successfully complete a specific task within the prototype or wireframe, providing valuable insights into the effectiveness and clarity of the design.
Pitfalls to avoid
- The goal isn’t to automatically optimise task related metrics but make sure they align with the goals of each section. User onboarding should take the time it needs to take to make sure the user understands the core mechanics and value of the product. This takes a little time.
- When evaluating the information architecture (IA) of the app, it’s crucial to pay attention to tree testing success rates and card sorting results. While these aren’t strictly metrics, this will provide a more informed design process and help create a foundation for more user satisfaction.
Development Phase:
During the development phase, usability testing metrics take centre stage. I strongly believe that user error rates; (Calculated by Number of errors / Total number of task attempts x 100) and System Usability Scale (SUS) scores are mission-critical. Nathan Thomas’ Usability Geek artcile “How To Use The System Usability Scale (SUS) To Evaluate The Usability Of Your Website” provides an indepth how-to guide on testing your product. These metrics provide valuable insights into how well users can interact with the app and complete key tasks. Accessibility testing metrics, such as conformance to accessibility guidelines, are also essential to ensure that the app is meeting standards and can be used by all.
Pitfalls to avoid
- If you’ve done the proper work with the previous phases, error rates will be low and SUS scores should be higher. BUT if error rates are happening, address them. Think of it like a product’s heart rate monitor beeping just a little too slowly.
- At this point, it’s crucial to be mindful of the sunk cost fallacy – the tendency to continue investing in a project because of the resources already put into it, even if it’s not delivering the desired results. If usability testing reveals significant issues, TrioLingo should be prepared to pivot or make major changes, rather than pressing forward with a suboptimal product.
Pre-launch Phase:
As TrioLingo prepares to launch its app, beta testing metrics become critical. User engagement metrics, such as session duration and feature usage, are mission-critical to assess how well the app engages users. User feedback and bug reports are also essential to identify and address any issues before launch. User acceptance testing (UAT) metrics, such as task completion rates and user satisfaction scores, are nice to have but can provide additional insights into the overall user experience.
Pitfalls to avoid
- While user engagement metrics like session duration and feature usage are important, focusing too heavily on these “vanity metrics” can be misleading. High session durations or feature usage do not necessarily translate to a satisfactory user experience or long-term user retention. TrioLingo should balance these metrics with other critical factors, such as task completion rates, user satisfaction scores, and qualitative feedback, to gain a comprehensive understanding of the app’s performance and user reception.
- By considering a holistic set of metrics and user insights, TrioLingo can make informed decisions and prioritise improvements that truly enhance the user experience.
Post-launch Phase:
Once the TrioLingo app is live, usage metrics become a higher priority. I believe that user engagement metrics, such as daily active users (DAU) and session duration, are mission-critical to assess user adoption and engagement. Conversion metrics, such as conversion rates for key user actions and funnel completion rates, are also essential to measure the app’s effectiveness in driving desired user behaviours. User satisfaction metrics, such as Net Promoter Score (NPS) and user reviews, are nice to have but can provide valuable insights into user sentiment and loyalty.
Pitfalls to avoid
- It’s important to emphasise that these metrics aren’t just for storytelling. They are critical tools for marrying product performance with user satisfaction. By tracking and analysing these metrics at each stage of product development, we can make data-driven decisions that optimise the user experience and drive business success.
Bringing it all together:
A crucial aspect of aligning metrics with your product development process is identifying your North Star metric. Our article “Measuring Success: finding your North Star Metric and the KPIs that support it” details the options available for measuring success. This metric represents a specific action you want your customers to complete that drives both adoption and revenue. It serves as a guiding light for your product strategy, helping you navigate the myriad of choices and maintain a clear direction.
When defining your North Star, consider the following questions:
How many active/returning users are taking this action?
What’s the depth of engagement required for users to complete this action?
How often does each user engage in this action? How quickly do users succeed in completing this action?
Turning back to TrioLing their North Star metric should be considered from three angles:
- The number of user actions needed to understand the key mechanics to have the highest chance of coming back.
- The above on a measurable timeline to help understand the LTV (Life Time Value)
Developing a North Star metric is a balancing act between key user actions and business needs. It requires a deep understanding of your users’ pain points, behaviours, and segmentation. Conducting thorough user research, both quantitative and qualitative, is essential to uncover these insights and inform your North Star metric.
Remember, your North Star metric is unique to your organisation and product. It should be developed collaboratively by those who lead product and business interests within your company, including founders and key stakeholders. Active management and focused coordination among product teams are crucial to maintaining and leveraging your North Star effectively.
Avoid the temptation to turn your product into a “feature farm” by chasing every stakeholder’s idea. Instead, let your North Star guide your product decisions, ensuring that you’re building a product that truly resonates with your users’ needs and drives business success. For TrioLingo, this might mean prioritising features and improvements that directly support course completion, such as personalised learning paths, gamification elements, or social features that encourage accountability and engagement.
By aligning your product development process with a well-defined, user-centric North Star metric, you can create a product strategy that shines brighter and delivers successful outcomes. For TrioLingo, focusing on course completion as their North Star can help them create a language learning app that stands out in a crowded market, driving user success and business growth.
My conclusion:
Aligning design-centric metrics with the product development process is not just about measuring the health of a product but also about guiding its direction. These metrics serve as a compass, helping teams navigate the landscape of product development and make informed decisions at every stage.
Aligning metrics with the product development process is not just a best practice; it’s a necessity for creating products that users will engage with and that achieve business objectives. By leveraging data-driven insights, guided by a well-defined North Star, and using them to inform product decisions, teams can create exceptional user experiences that drive engagement, conversion, and loyalty.
If you want to take your product development process to the next level, consider investing in ideation and research workshops. These workshops bring together experts in design, research, and strategy to rapidly generate and test innovative ideas, ensuring that your product stays true to its North Star. By validating concepts early and often, teams can save time and resources in the long run and ensure that they are building products that truly resonate with users and drive business success.
With thanks to Campaign Creators, Greg Rakozy, LinkedIn Sales Solutions, Photo by David Travis, Markus Winkler, Marek Piwnicki on Unsplash for the photos ❤️