Transforming Performance Metrics from Data to Stories

understanding the problem

The core issue was the old speed score system’s unreliability and its lack of actionable insights for users. We recognized the potential of Google’s Core Web Vitals, a trusted and reliable metric, to provide a more accurate measure of an online store’s performance. However, the challenge was to present these technical metrics in a way that was accessible and actionable for everyday merchants.

The collection of feedback above represents the problem at hand: the need for a reliable, understandable, and actionable performance metric system for online store merchants.

And some also I asked myself, Why did my speed score go from 45 to 37? Because I didn’t change in on September 8th. On that day I didn’t do anything.
‍
– Research Participant #2

Research

A significant amount of research was conducted to gain a deeper understanding of the problem. This began with a competitor analysis, which allowed us to understand how other platforms were tackling similar issues and identify potential areas of improvement. We then conducted an extensive survey targeting merchants who had previously interacted with the speed score.

To further enrich our understanding, we conducted 12 interviews with select merchants from the survey. These conversations allowed us to delve deeper into their pain points and understand their needs on a more personal level. These interviews provided us with a broader perspective on the problem and helped us identify common themes and issues across different types of users.

Understanding user needs

The user personas for this project were non-tech merchants, some-tech knowledge merchants, and developers. Their needs included the need for better understanding of what impacts their performance, the need for more help and handholding, and the preference for insights from multiple directions instead of just one.

Ideation and design

The ideation phase began with the creation of low-fidelity designs. These initial sketches served as conversation starters within the team, sparking discussions about potential solutions and design directions. This early collaboration ensured that all team members, including developers and data scientists, were involved in the design process from the start, fostering a shared understanding and ownership of the project.

To further explore potential solutions, we prototyped a few ideas based on what we observed from our competitor analysis. These prototypes were not meant to be final solutions but rather tools to facilitate user interviews. They allowed us to present concrete ideas to the merchants and gather their feedback, preferences, and reactions.

We then embarked on a series of quick exploration and feedback cycles (with the developers and data scientists of the team), each lasting one week. During these cycles, we iterated on our prototypes based on the user feedback we received, while also keeping in mind what they preferred from the competitor’s solutions. This iterative process allowed us to refine our ideas continually and move closer to a design that addressed the users’ needs effectively.

The key features of our final design solution were heavily influenced by our research findings and the feedback we received during the ideation phase. We aimed to explain Core Web Vitals in a way that was more comprehensive and accessible than anywhere else on the internet.

I designed the interface to allow merchants to drill down into each metric, enabling them to see when a problem started, what caused it, and where it was happening. Recognizing that many of our users wouldn’t have enough traffic to have confident results, we also included explanations of low-confidence data and guided them towards more fitting data views.

Final designs

I put a lot of effort into making sure that the main component of the dashboard, the 3 cards look great on any sizes of screens and the interactions are flawless.

I deeply care about the developers’ happiness and I always make sure to deliver documentation that covers all use cases and small details.

Outcome and reflections

The project resulted in a user-friendly interface that made complex performance data accessible for merchants, improving their online store’s performance and user experience.

Despite challenges such as dealing with a very opinionated PM and designing for edge-cases, the project provided valuable lessons. It enhanced skills in team engagement, rapid iteration based on new data points, and maintaining focus on initial problems and user needs.

In retrospect, a more collaborative approach, particularly in analyzing interview results, would be adopted in future projects.

‍