Planning and Metrics

Scrum metrics and reporting – measure what you manage


Posted by Ilan Goldstein on August 29, 2011

Youā€™ve introduced Scrum and your team is out of the blocks, sprinting away! Things are going pretty well until some wise guy from senior management comes up to you and says something along the lines of: ā€œSooo, this whole Scrum thing sounds great in theory but what metrics are you going to use to actually demonstrate to us how effective it really is…?ā€

Like it or not, people just love to measure and compare and because of this, when it comes to avoiding metrics, you can run, but you canā€™t hide. In this article, I will offer you some suggestions as to what metrics actually matter when it comes to implementing Scrum.

Types of metrics

The most important piece of advice that I can offer when it comes to metrics is to only use metrics for good and not for evil. Considering that there arenā€™t any readily available global definitions for what a good and an evil metric are, I have come up with my own:

Good metric ā€“ Used as a signal to help the team identify roughly where things are at and more importantly, as a guide to help the team inspect and adapt its processes to improve over time.

Evil metric ā€“ Used as an inflexible indicator for micro-managing an individualā€™s performance over time and more importantly, for ā€˜beatingā€™ people up and killing morale.

I look at grouping metrics into two main categories:

  • Metrics for Scrum projects (the focus of this article)
  • Metrics for broader Scrum rollouts (the focus for another article)

Four meaningful metrics

The following sections will step you through a selection of four project related metrics that I find particularly helpful including:

  • Sprint burndown
  • Enhanced release burndown
  • Sprint interference
  • Remedial focus

Sprint burndown

The sprint burndown is a forecasting metric to assist in tracking progress throughout the current sprint.

How is it generated?

  1. For each day in a sprint, plot the sum of the ā€˜remaining timesā€™ for all tasks in the sprint backlog.
  2. Draw a connecting line between the current dayā€™s total and the previous dayā€™s
A sprint burndown chart - Scrum Metrics that Matter by AxisAgile

Figure 1 – A sprint burndown chart, updated on a daily basis.

When is it generated?
The sprint burndown is generated at the end of each day of a sprint (excluding the final day, which is dedicated to the sprint review, retrospective, and planning for the subsequent sprint).

What is it telling you?
The sprint burndown metric acts as a daily gauge for the Scrum team to help manage its workflow and gauge progress.

If the chart is trending behind schedule, it could be reflecting the fact that:

  • New tasks were added to the sprint backlog (that werenā€™t anticipated during sprint planning).
  • Some of the task estimations were incorrect.
  • Team members had taken some unplanned time off.
  • Impediments had hampered progress.
Sprint burndown chart behind schedule - Scrum Metrics that Matter by AxisAgile

Figure 2 – The team is clearly behind schedule; time to speak to the product owner about possibly decreasing scope.

Of course, it is possible that all four factors had come into play causing the now expected delay.

Following sprint planning, many teams will draw a straight, diagonal (theoretical) line from the top of the y-axis values to end of the x-axis values and this is used as a benchmark for the actual burndown line. I warn against using this approach as it can easily create an inaccurate perception of progress. The problem with this line is that sprint progress will rarely mirror the theoretical line on a day-to-day basis. Many sprint burndown lines will actually burn up for the first few days due to new discoveries before beginning itā€™s downward descent as the team gathers momentum. By including the theoretical line, a curious stakeholder may get the misleading impression that the team has fallen behind after only just one or two days.

How can you act on it?
If the sprint burndown clearly indicates that the team is not going to reach the sprint goal then apart from doing everything in your power to help remove any impediments, it should prompt a discussion with the product owner to assess whether any scope can be removed. If the slip is due to inaccurate estimating of tasks, it can be helpful to analyze why the estimates were wrong to try and improve the sprint planning accuracy for the next sprint.

The team is clearly behind schedule; time to speak to the product owner about possibly decreasing scope.

Sprint burndown charts can also paint a rosier picture (believe it or not) if they trend steeply towards an early completion of the sprint backlog. If this is the case, the burndown should prompt the product owner to prepare the next sprint-ready product backlog items for additional consumption prior to the forthcoming sprint planning session.

A sprint burndown chart ahead of schedule - Scrum Metrics that Matter by AxisAgile

Figure 3 – The team is clearly ahead of schedule; time to speak to the product owner about adding scope.

Enhanced release burndown

The enhanced release burndown is inspired by Mike Cohnā€™s ā€˜Alternative Scrum Release Burndown Chartā€™.

How is it generated?

  1. For each sprint, plot the sum of the ā€˜remaining pointsā€™ for all product backlog items in the product backlog designated for the next release.
  2. Draw a trend line relating to the data points in step 1.
  3. For each sprint, plot (as negative y-axis values) the sum of the story points for any product backlog items added to the product backlog after the start of the project (if applicable).
  4. Draw another trend line that relates to the data points in step 3.

When is this generated?
The enhanced release burndown metric is generated at the end of every sprint.

What is it telling you?
This metric signals what the development teamā€™s rate of progress is relative to the scopeā€™s rate of change. The point where the two trend lines (hopefully) intersect indicates roughly how many sprints will be required to complete the release. If the trend lines run parallel to each other (or diverge), it is an ominous indication that the release will theoretically never see the light of day.

An enhanced release burndown chart - Scrum Metrics that Matter by AxisAgile

Figure 4 – An enhanced burndown chart, updated after each sprint.

How can you act on it?
If the two trend lines do not intersect or the expected release duration is intolerable, then either the rate of progress needs to increase (by improving practices and / or removing impediments) or the scope needs to be reduced.

An enhanced release burndown chart that is looking ominous - Scrum Metrics that Matter by AxisAgile

Figure 5 – Ominous signs that this release might never see the light of day. Better improve practices, remove impediments, or decrease scope.

Sprint interference

Sprint interference is a productivity metric to assist teams with their sprint capacity planning.

How is it generated?

  1. For each sprint, plot the sum of the time spent by any of the developers for any non-sprint backlog tasks.
  2. Draw a trend line that relates to the data points in step 1.
A sprint interference graph used to plan the team's next sprint capacity - Scrum Metrics that Matter by AxisAgile

Figure 6 – Be sure to note the trend in this graph to assist in estimating your teamā€™s capacity at the next sprint planning session.

Be sure to note the trend in this graph to assist in estimating your team’s capacity at the next sprint planning session.

When is this generated?
The sprint interference metric is generated during sprint planning.

What is it telling you?
By providing visibility on the time spent handling historical sprint disruptions, this metric will help you to estimate the potential sprint capacity for the forthcoming sprint (the amount of time that the development team should allocate to sprint backlog tasks). This is especially helpful if you have adopted commitment-based sprint planning.

How can you act on it?
In any given sprint, there will be a range of external organizational disruptions that simply canā€™t be avoided. This metric assists in quantifying these disruptions and can also indirectly help to identify what are unavoidable disruptions (such as company meetings) and what are avoidable impediments (such as constantly having to fix inadequate equipment).

Remedial focus

How is it generated?

  1. For each sprint, plot the total velocity (the sum of the points for all product backlog items including both new functionality and bugs.)
  2. For each sprint, plot the sum of the points for all bug related work.
  3. Draw a trend line that relates to the data points in step 2.
A remedial focus graph showing improving quality over a number of sprints - Scrum Metrics that Matter by AxisAgile

Figure 7 – This indicates that quality is improving. The effort spent on bug-fixing is going down while velocity remains consistent.

When is this generated?
The remedial focus metric is generated at the end of each sprint.

What is it telling you?
This metric monitors the fluctuations in product quality by measuring the percentage of each sprint that is spent working on bugs as opposed to new functional requirements.

In addition, by quantifying the make up of the ā€˜total velocityā€™, additional insight can be garnered. For example, the total velocity may be trending upwards, which, on the surface would typically indicate positive improvement. However, if the amount of bug related work is also trending upwards this is an indication that the level of quality is slipping. As such, the increase in total velocity could in fact be indicating that the team is just getting faster at fixing its own bugs ā€“ somewhat of a back-handed compliment…

A remedial focus graph showing improving velocity but decreasing quality over a number of sprints - Scrum Metrics that Matter by AxisAgile

Figure 8 – Even though velocity is increasing, itā€™s not all good news, as quality is decreasing. Time to revisit the definition of done.

How can we act on it?
If the time spent on bugs isnā€™t trending downwards, it is a clear indication that the level of inherent quality is insufficient. This should prompt the Scrum team to revisit the definition of done to tighten up the quality requirements.

Beware of analysis paralysis

These four metrics form a small subset of the potential metrics that can be derived and utilized during a Scrum project. But be warned; incessantly generating metrics (just because you can), may cause you to contract an illness that Scrum on the whole tries to combat – analysis paralysis.

Finally, it is vitally important to reinforce the point that metrics need to be handled with care and used for good and not for evil. Use metrics as signals to help your team improve their practices and definitely not for micro-managing and measuring individual performance.

If you liked this article, you can:
Subscribe to this RSS feed!

Find out more about Scrum related metrics by taking one of our CSM training courses.

SOME OF OUR GREAT CLIENTS

testimonials

Sign up to receive our eNewsletter

Google Rating
5.0
Based on 309 reviews
js_loader