Prioritize Efforts By Exploring What Outcomes Matter To Users

Impact

Insights from this study are used as one of the data sources to prioritise upcoming opportunities in the domain roadmap.

image

Context

  1. Within Sonar products (SonarCloud, SonarQube, SonarLint), we emphasize the Clean as You Code methodology as one of the core offerings to help users evaluate and work on the quality and security of their code.
  2. As part of the Clean as You Code methodology, we usually emphasize their following key benefits:
    1. Systematically achieve the state of Clean Code (with the right code quality standard)
    2. Reduce technical debt
    3. Enable Developers to take ownership of their code
    4. Avoid the stress of resolving legacy issues in code
    5. Slowly remediate past issues as Developers work on old code to make new edits
  3. However, we do not have data indicating if these benefits we offer align with what users expect as a result of using Sonar products.

Why

  1. This level of data can help us evaluate if we are investing the right effort to deliver on the right outcomes
    1. Are we failing to deliver on some outcomes that are perceived as more important?
    2. Are we putting effort on desired outcomes that are perceived as less important?
  2. How can we we better connect the Clean as You Code methodology in our products to the users desired outcomes?
    1. Is there a need to refine the scope of the methodology?

What

  1. Identify the desired outcomes that Developers expect from Sonar as a solution
  2. Assess each outcome with respect to:
    1. Importance: How important is the outcome to the Developer
    2. Satisfaction: How satisfied is the Developer with Sonar as a solution helping them achieve the desired outcome
  3. Based on the importance and satisfaction scores, each outcome will be classified into one of the three following categories
    1. Underserved: Users are comparatively less satisfied with the outcome than how important the outcome is
    2. Balanced: The user satisfaction is aligned with the importance of the outcome
    3. Overserved: Users are comparatively more satisfied with the outcome than how important the outcome is

How

Research was split into three phases

  1. Phase 1: Generate outcomes that Developers expect from Sonar as a solution
    1. One hour workshop with a few Developers within Sonar
    2. Time taken: 1 week
  2. Phase 2: Evaluate the outcomes from Phase 1 on a quantitative basis
    1. Two closed card sorting exercises through UserTesting in five rounds. Each round took a week, had the same setup and targeted 30 Sonar users
    2. Time taken: 5 weeks
  3. Phase 3: Analyze the quantitative data to classify each outcome as underserved, balanced or overserved
    1. Generate importance and satisfaction score for each outcome
    2. Visualize the outcomes using the importance and satisfaction scores as coordinates
    3. Time taken: 1 week

What we learned

  1. We could identify 30 desired outcomes during Phase 1, thanks to our focus group with the Developers
    1. We classified these 30 outcomes into 10 different jobs to be done
    2. 19 of these 30 outcomes were identified by our domain PM and Engineers as those that are addressed by Clean as You Code methodology
  2. After gathering quantitative data through card sorting exercises, calculating importance and satisfaction scores for each outcome - we end up with this visualization:
image
💡

How to read the graph: The diagonal line indicates that the satisfaction to importance ratio is 1. In decreasing order of importance, opportunities related to underserved outcomes need to be prioritized higher than the rest. Outcomes above the line are overserved and those below the line are underserved.

  1. Except for a couple of outliers, most outcomes are concentrated around the reference line indicating a healthy level of balance
  2. 6 out of 19 Clean as You Code outcomes stand out with high perceived importance - indicating the scope of current methodology aligns with Developers expecations to a fair extent
  3. On the other hand, three outcomes related to Clean as You Code witnessed some of the least importance scores (Outcome #9, 17 and 27) which uncovered opportunities to prioritize in the roadmap

Limitations & notes

  1. The list of outcomes we identified is non-exhaustive
  2. This study is targeted towards one of the personas we define in our teams

Team: Joseph Duteil (UX Designer), Silpa Vipparti (UX Researcher)