Blog

Scrum metrics and reporting – measure what you manage

6

You’ve introduced Scrum and your team is out of the blocks, sprinting away! Things are going pretty well until some wise guy from senior management comes up to you and says something along the lines of: “Sooo, this whole Scrum thing sounds great in theory but what metrics are you going to use to actually demonstrate to us how effective it really is…?”

Like it or not, people just love to measure and compare and because of this, when it comes to avoiding metrics, you can run, but you can’t hide. In this article, I will offer you some suggestions as to what metrics actually matter when it comes to implementing Scrum.

Types of metrics

The most important piece of advice that I can offer when it comes to metrics is to only use metrics for good and not for evil. Considering that there aren’t any readily available global definitions for what a good and an evil metric are, I have come up with my own:

Good metric – Used as a signal to help the team identify roughly where things are at and more importantly, as a guide to help the team inspect and adapt its processes to improve over time.

Evil metric – Used as an inflexible indicator for micro-managing an individual’s performance over time and more importantly, for ‘beating’ people up and killing morale.

I look at grouping metrics into two main categories:

  • Metrics for Scrum projects (the focus of this article)
  • Metrics for broader Scrum rollouts (the focus for another article)

Four meaningful metrics

The following sections will step you through a selection of four project related metrics that I find particularly helpful including:

  • Sprint burndown
  • Enhanced release burndown
  • Sprint interference
  • Remedial focus

Sprint burndown

The sprint burndown is a forecasting metric to assist in tracking progress throughout the current sprint.

How is it generated?

  1. For each day in a sprint, plot the sum of the ‘remaining times’ for all tasks in the sprint backlog.
  2. Draw a connecting line between the current day’s total and the previous day’s
A sprint burndown chart - Scrum Metrics that Matter by AxisAgile

Figure 1 – A sprint burndown chart, updated on a daily basis.

When is it generated?
The sprint burndown is generated at the end of each day of a sprint (excluding the final day, which is dedicated to the sprint review, retrospective, and planning for the subsequent sprint).

What is it telling you?
The sprint burndown metric acts as a daily gauge for the Scrum team to help manage its workflow and gauge progress.

If the chart is trending behind schedule, it could be reflecting the fact that:

  • New tasks were added to the sprint backlog (that weren’t anticipated during sprint planning).
  • Some of the task estimations were incorrect.
  • Team members had taken some unplanned time off.
  • Impediments had hampered progress.
Sprint burndown chart behind schedule - Scrum Metrics that Matter by AxisAgile

Figure 2 – The team is clearly behind schedule; time to speak to the product owner about possibly decreasing scope.

Of course, it is possible that all four factors had come into play causing the now expected delay.

Following sprint planning, many teams will draw a straight, diagonal (theoretical) line from the top of the y-axis values to end of the x-axis values and this is used as a benchmark for the actual burndown line. I warn against using this approach as it can easily create an inaccurate perception of progress. The problem with this line is that sprint progress will rarely mirror the theoretical line on a day-to-day basis. Many sprint burndown lines will actually burn up for the first few days due to new discoveries before beginning it’s downward descent as the team gathers momentum. By including the theoretical line, a curious stakeholder may get the misleading impression that the team has fallen behind after only just one or two days.

How can you act on it?
If the sprint burndown clearly indicates that the team is not going to reach the sprint goal then apart from doing everything in your power to help remove any impediments, it should prompt a discussion with the product owner to assess whether any scope can be removed. If the slip is due to inaccurate estimating of tasks, it can be helpful to analyze why the estimates were wrong to try and improve the sprint planning accuracy for the next sprint.

The team is clearly behind schedule; time to speak to the product owner about possibly decreasing scope.

Sprint burndown charts can also paint a rosier picture (believe it or not) if they trend steeply towards an early completion of the sprint backlog. If this is the case, the burndown should prompt the product owner to prepare the next sprint-ready product backlog items for additional consumption prior to the forthcoming sprint planning session.

A sprint burndown chart ahead of schedule - Scrum Metrics that Matter by AxisAgile

Figure 3 – The team is clearly ahead of schedule; time to speak to the product owner about adding scope.

Enhanced release burndown

The enhanced release burndown is inspired by Mike Cohn’s ‘Alternative Scrum Release Burndown Chart’.

How is it generated?

  1. For each sprint, plot the sum of the ‘remaining points’ for all product backlog items in the product backlog designated for the next release.
  2. Draw a trend line relating to the data points in step 1.
  3. For each sprint, plot (as negative y-axis values) the sum of the story points for any product backlog items added to the product backlog after the start of the project (if applicable).
  4. Draw another trend line that relates to the data points in step 3.

When is this generated?
The enhanced release burndown metric is generated at the end of every sprint.

What is it telling you?
This metric signals what the development team’s rate of progress is relative to the scope’s rate of change. The point where the two trend lines (hopefully) intersect indicates roughly how many sprints will be required to complete the release. If the trend lines run parallel to each other (or diverge), it is an ominous indication that the release will theoretically never see the light of day.

An enhanced release burndown chart - Scrum Metrics that Matter by AxisAgile

Figure 4 – An enhanced burndown chart, updated after each sprint.

How can you act on it?
If the two trend lines do not intersect or the expected release duration is intolerable, then either the rate of progress needs to increase (by improving practices and / or removing impediments) or the scope needs to be reduced.

An enhanced release burndown chart that is looking ominous - Scrum Metrics that Matter by AxisAgile

Figure 5 – Ominous signs that this release might never see the light of day. Better improve practices, remove impediments, or decrease scope.

Sprint interference

Sprint interference is a productivity metric to assist teams with their sprint capacity planning.

How is it generated?

  1. For each sprint, plot the sum of the time spent by any of the developers for any non-sprint backlog tasks.
  2. Draw a trend line that relates to the data points in step 1.
A sprint interference graph used to plan the team's next sprint capacity - Scrum Metrics that Matter by AxisAgile

Figure 6 – Be sure to note the trend in this graph to assist in estimating your team’s capacity at the next sprint planning session.

Be sure to note the trend in this graph to assist in estimating your team’s capacity at the next sprint planning session.

When is this generated?
The sprint interference metric is generated during sprint planning.

What is it telling you?
By providing visibility on the time spent handling historical sprint disruptions, this metric will help you to estimate the potential sprint capacity for the forthcoming sprint (the amount of time that the development team should allocate to sprint backlog tasks). This is especially helpful if you have adopted commitment-based sprint planning.

How can you act on it?
In any given sprint, there will be a range of external organizational disruptions that simply can’t be avoided. This metric assists in quantifying these disruptions and can also indirectly help to identify what are unavoidable disruptions (such as company meetings) and what are avoidable impediments (such as constantly having to fix inadequate equipment).

Remedial focus

How is it generated?

  1. For each sprint, plot the total velocity (the sum of the points for all product backlog items including both new functionality and bugs.)
  2. For each sprint, plot the sum of the points for all bug related work.
  3. Draw a trend line that relates to the data points in step 2.
A remedial focus graph showing improving quality over a number of sprints - Scrum Metrics that Matter by AxisAgile

Figure 7 – This indicates that quality is improving. The effort spent on bug-fixing is going down while velocity remains consistent.

When is this generated?
The remedial focus metric is generated at the end of each sprint.

What is it telling you?
This metric monitors the fluctuations in product quality by measuring the percentage of each sprint that is spent working on bugs as opposed to new functional requirements.

In addition, by quantifying the make up of the ‘total velocity’, additional insight can be garnered. For example, the total velocity may be trending upwards, which, on the surface would typically indicate positive improvement. However, if the amount of bug related work is also trending upwards this is an indication that the level of quality is slipping. As such, the increase in total velocity could in fact be indicating that the team is just getting faster at fixing its own bugs – somewhat of a back-handed compliment…

A remedial focus graph showing improving velocity but decreasing quality over a number of sprints - Scrum Metrics that Matter by AxisAgile

Figure 8 – Even though velocity is increasing, it’s not all good news, as quality is decreasing. Time to revisit the definition of done.

How can we act on it?
If the time spent on bugs isn’t trending downwards, it is a clear indication that the level of inherent quality is insufficient. This should prompt the Scrum team to revisit the definition of done to tighten up the quality requirements.

Beware of analysis paralysis

These four metrics form a small subset of the potential metrics that can be derived and utilized during a Scrum project. But be warned; incessantly generating metrics (just because you can), may cause you to contract an illness that Scrum on the whole tries to combat – analysis paralysis.

Finally, it is vitally important to reinforce the point that metrics need to be handled with care and used for good and not for evil. Use metrics as signals to help your team improve their practices and definitely not for micro-managing and measuring individual performance.

Have you got your own metrics that you’d like to share? What did you do when you got the tap on the shoulder asking you to report some metrics? Let us know in the comments below.

If you liked this article, you can:
Subscribe to this RSS feed!


About the author

Ilan

Ilan

Director at AxisAgile

I’m Ilan Goldstein and I’m a director here at AxisAgile. I’m a Certified Scrum Trainer and the author of Scrum Shortcuts Without Cutting Corners.

Find out more about me or get in contact with me by using the social buttons. I also respond to comments below.

Ilan

Ilan

Ilan

Latest posts by Ilan (see all)

  1. Prabhakar Karve
    Prabhakar Karve07-07-2013

    I liked the structure of explanation of each metric in terms of,
    How is it generated?
    When is this generated?
    What is it telling you?
    How can we act on it?

    As regards good and bad metrics, I have found a different yardstick quite useful.
    – Identify the stakeholders
    – find out what is at stake for them
    – design metrics to take care of their interests
    – use transparency to share the metrics
    – With those who can inspect and adapt
    – To improve their practices
    – Which in turn would take care of stakeholders’ interests

    Does it make sense?
    Karve

  2. Srinath
    Srinath08-26-2013

    Nice article .. Have a few questions on some of the metrics :

    1. Enhanced Release Burndown – Would it not be difficult to come out with the remaining Story points early in a project ? The Product backlog goes through grooming sessions (refinement sessions) prior to each planning meeting in the course of which the number of story points could drastically change (both ways) – which could impact the final delivery dates. Would it then be right to come out with a tentative release date after just a few iterations (with ref to Fig 4 and Fig 5) ?

    2. Sprint interference – this assumes that Non sprint tasks are captured as part of the metrics by the Scrum master. I wonder if this is captured, and if so, would give us a correct picture?

    • Ilan
      Ilan09-14-2013

      Hi Srinath,

      1) You are able to re-forecast the evolving, emergent backlog at the end of every sprint by dividing the total points of the backlog by the new current velocity (I personally like to use a rolling average of the last three sprints).

      2) I recommend that each developer captures their Sprint Interference rather than just the ScrumMaster. It will at least be more accurate than guessing capacity or locking in an arbitrary number e.g. 6 hours.

  3. Jayashree Nagaraj
    Jayashree Nagaraj09-13-2013

    Hello IIan – I came to your blog to read about your book and found this post relevant to what I am trying to figure out right now at work. I play a scrum master role and am re-looking at the type of metrics I should be capturing. Some follow up questions
    – is it useful to track when a story or task takes longer or less time to implement than planned? How can we use this information?
    – if a story could not be completed in sprint 1 and had a SPE of 8 and is going to be taken up in sprint 2, should I assume the same SPE for the story in sprint 2 or should we re-calculate effective SPE for sprint 2? I am worried about how this could skew the info/graph on velocity of team in a sprint and across sprints.

    Would appreciate your insight. Thanks.

    • Ilan
      Ilan09-14-2013

      Hi Jayashree,

      1) Tracking estimation accuracy might be useful if you use the information to help improve knowledge moving forward. For example, if an estimate was inaccurate due to a lack of understanding of a particular area of the code, this can be useful information that may improve estimates and more importantly, understanding of the code moving forward.

      2) I count the ALL points for a story in the sprint that the story gets completed. Indeed this will skew the immediate velocity and that is why I like using a rolling-average of the last 3 sprints to level out such discrepancies.

Leave a Reply