SonarCloud Annual Benchmarking Program

Impact

The experience to upgrade from a free plan to paid plan in SonarCloud improved in by 56% in terms of usability!

image

Why

  1. When I joined as the first UX Researcher, I realised we did not have a baseline for how existing users perceive SonarCloud concerning overall usability and how they navigate to complete different key journeys
  2. Establishing metrics to evaluate these key journeys on a periodic basis can help us get a bird’s eye view of what’s working, what’s not and what needs improvement

What

  1. Benchmarking is a common practice used to evaluate product performance periodically through the UX lens
  2. The plan is to evaluate the same key journeys over a set of metrics on a fixed frequency to track performance over time

How

  1. We discussed with the Product Manager and Designer to identify the six key tasks to evaluate in SonarCloud
  2. Each task will be evaluated on the following metrics:
    1. Completion rate
    2. Error count
    3. Time taken to complete the task
    4. Perceived user effort
    5. Severity
  3. Inspired from GitLab’s approach, we adapted their method to calculate a single grade for each task based on the five metrics

Some important learnings

  1. When the study was first conducted in 2022
    1. The journey for a user to upgrade from free plan to paid plan was the poorest performing task
      1. This happened because the path to upgrade did not have an intuitive starting point
      2. Users spent about a minute figuring out how to get started to upgrade (that’s a lot!)
    2. Once user saw the option to upgrade was under ‘Administration → Billing’, there wasn’t much information on the page about what the current plan includes or what the benefits of paid plan are
  2. With these findings, our product team members worked to iterate on the experience by
    1. Moving the option to upgrade to the main navigation list, and labelling it ‘Billing and upgrade’
    2. Introduced a slider UI for users to explore different tiers of paid plans and understand the benefits
    3. Clearly highlighting the users current plan, making it easy for users to compare with what they want to pick
  3. Eventually, in 2023 when we conducted the same study again - Upgrade experience turned out to be the best performing task with a score of 92/100!

Team: Martin Bednorz (Product Manager), Laura Warcrenier (UX Designer), Silpa Vipparti (UX Researcher), Zina Alaswad (UX Researcher)