How to prioritize bugs and feature requests in real-time

John Micah Reid
5 min readSep 4, 2024

--

Reading Product Manager job descriptions, you’d be forgiven for thinking that the job is just sitting around a conference table with post-it notes and whiteboards, ‘defining strategy’ and especially ‘prioritizing roadmaps’.

this would never happen in real life right?

For the uninitiated, this is a regular exercise (e.g. quarterly) where you gather a long list of ideas from stakeholders and users, then sort them into prioritized buckets such as Now/Next/Later using a framework like Reach-Impact-Confidence-Effort. Here’s one of my previous articles if you want to learn more:

However, this is actually only a small part of the prioritization work. Most of the time you’ll be receiving a constant stream of ad-hoc incoming issues, for example:

  • Crash reports are up 20% on Android after the new release
  • Marketing needs a new demo built
  • Several users have asked for a new download file format

You need to decide what to do with each of them one at a time. This is a harder problem than the quarterly roadmap exercise because you can’t just wait to gather them in a batch and compare them against each other — you don’t know what else might come up in the near future. You need to prioritize in “real-time” (days) rather than in “batch” (months).

(Tangent: there’s a whole field of research around this called stopping theory with intriguing names like the secretary problem, marriage problem and sultan’s dowry problem where you try to pick the best option from a sequence without knowing how many other options you’ll see)

It’s tempting to follow one of the extreme approaches:

  • Say no to everything — you stick to your guns and never fix critical bugs or address small, easy feature requests that end up having a disproportionate impact on user satisfaction.
  • Take on all new issues— you add half-baked fixes to the product that clutter it up and never make progress on your main objectives.

Most PMs muddle around in the middle, saying yes to some requests and no to others depending on the day of the week. Those around you get frustrated by your exercise of petty authority, which can lead to distrust.

In this article we’ll discuss an alternative system for real-time prioritization. I’ve used this successfully in my current role to speed up the process and provide more transparency to everyone around me.

The system

The idea is to score each incoming issue by several criteria, then prioritize based on the final score.

The criteria are as follows:

  1. Is it a bug or feature request? (bug ⬆️)
  2. Are multiple users reporting an issue? (more users ⬆️)
  3. What % of our user base is potentially affected, as measured by the % who use this feature? (more users ⬆️)
  4. What is the business impact of the issue? (more severe ⬆️)
  5. What is the estimated time to implement a change? (lower and more certain ⬆️)

These generate a score between 0 and 100, which result in the following priority levels:

  • 70–100 → 🟥 Highest: Take into current sprint
  • 40–70 → 🟧 High: Estimate for next sprint
  • 20–40 → 🟨 Low/Medium: Put in backlog
  • 0–20 → 🟦: No action

I’ve defined the exact options and weights for each question inside a Google Sheets calculator which you can try here (remember to make a copy first so you can edit it):

The scoring system has a couple of advantages:

  • Transparent: the reasons why a ticket is prioritized or deprioritized are clearly shown by the weighting of different criteria — this will help you to explain to stakeholders why you will/won’t be working on their request
okay this was unrelated but hilarious
  • Reproducible & Consistent: the process can be followed by anyone, not just members of the product team, and the answers should be the same each time
  • Fast: No need to spend time debating about where a given issue fits in

Notes on using it

In terms of scheduling, you’ll need to reserve some capacity each sprint to actually work on these issues — 20% of engineering time is a good benchmark. If nothing more important comes up, you can use that extra time to work on tech debt and prevent your engineers from losing too much hair.

Some of the questions require more information to score e.g. for Question 3 you can link to a feature usage dashboard, and Question 5 might require getting an estimate from an engineer.

You’ll also want to adapt the questions and weighting for your industry and use-case. For example in B2B, you might want to weight incoming tickets from the top 20% of your customers more highly.

Worked Examples

Let’s go through a few of those previous examples to see how they end up getting prioritized — you can follow along on the sheet if you want to see the exact calculations:

Crash reports are up 20% on Android after the new release

  • Bug: +15
  • Multiple users reporting an issue: +15
  • 20-50% of userbase affected: +20
  • Impact is High (data loss): +30
  • Time to fix is Unknown: 0

The overall score here is 80, which means it is Highest Priority and you should work on it in your current sprint.

Marketing needs a new demo built

  • Feature: 0
  • No users reporting an issue: 0
  • 10–20% of userbase affected: +10 (assuming that’s the number of prospects who will see it)
  • Impact is low (cosmetic): 0
  • Time to implement is more than a week: 0

The overall score is 10, meaning you do not even put it in your backlog. This might sound harsh, but there are probably better ways to spend your team’s time.

Several users have asked for a new download file format

  • Feature: 0
  • Multiple users reporting an issue: +15
  • <10% of users affected: 0
  • Impact is medium (workaround required): +15
  • Time to implement is less than a day: +20

In this case the score is 50, meaning you should estimate it for the next sprint.

Closing Thoughts

You’ll notice that some scores result in not taking any action and essentially ignoring the issue 😱 — this might seem harsh but there’s no point putting something in the backlog that you’ll never work on because it’s so low priority. This is exactly the point of prioritization: choosing what you’re going to focus on so you can actually make progress.

Using this framework you can prioritize incoming issues one-at-a-time, without knowing the relative importance of everything else. This is a useful counterpoint to the batch-style prioritization that is more commonly known, because you are trading a bit of accuracy for a quicker decision. It’s also really helpful for staying flexible when your best-made quarterly roadmaps become out of date as they encounter reality.

--

--

John Micah Reid
John Micah Reid

Written by John Micah Reid

Technical product manager, interested in telling stories through data. https://www.linkedin.com/in/johnmicahreid/

No responses yet