I sometimes hear industry colleagues comment that their project team is, “About 85% agile” or they might say something like, “We are using Scrum to about 50% capacity. The typical question that goes through my head when I hear comments such as these is, “what the hell does that mean?”
Let’s take a look at the first point: “We’re 85% agile.” Really? So does that mean that if you do things a little bit differently, perhaps start implementing a few more practices then you will soon be 100% agile? Wow, you can’t do any better than 100% so when you hit this ‘auspicious’ milestone then I suppose that there is nothing more to do except maintain what you’re doing right? Wrong! I’ve said it before but you shouldn’t need me to convince you that 100% agile perfection is never attainable! There is always something that can be done better and considering that Scrum is fundamentally about continuous improvement, how can perfection ever be achieved? Classifying your ‘agility’ in terms of percentages just doesn’t make sense.
Humans love to measure
The obsession with measurement possibly starts for us when that first pencil mark is drawn on the wall by our proud parents measuring how tall we’re growing. Humans love to score and we love to assess our progress for a multitude of reasons. This isn’t a bad thing if done for the right reasons such as gauging continual process and measuring what Scrum was able to deliver. Assessment can also be a great team motivator to give everyone a sense of forward motion and progress in the same way that a belt system works in martial arts. Sure, to some, the belt is more important than the art, however, to most, the belt indicates that they have improved and been recognized for their hard work and that isn’t a bad thing in my book.
There are two primary reasons why you might find it necessary to quantify the success of your Scrum rollout:
- To decide whether you should continue with Scrum.
- To assess your team’s progress along their Scrum journey.
Should we continue?
So how do you gauge whether your initial Scrum rollout has been successful? Well you could rely on a bunch of numbers on a spreadsheet comparing your pre and post-Scrum projects but I for one don’t like using this approach (at least not in isolation) primarily because Scrum is not a mechanical process. It is so reliant on people and culture that even with fantastic quantitative results, the introduction of Scrum may have caused such upheaval that too many people have become unhappy and that is not good for Scrum’s long-term survival.
There is an approach that I particularly like that helps to enrich any quantitative feedback and that is a simple subjective, collaborative survey that I first read about in Scrum trainer, Gabrielle Benefield’s paper. To gauge the effectiveness of their pioneering global Scrum rollout at Yahoo, Benefield and Deemer utilized a simple survey based on the following six criteria.
Tracking Yahoo!’s Scrum Rollout
- How much the team got done in 30 days
- Clarity of goals / what the team was supposed to deliver
- Collaboration and cooperation within the team
- Business value of what the team produced in 30 days
- Amount of time wasted / work thrown out / cycles misused
- Overall quality and ‘rightness’ of what the team produced
Everyone involved in the survey had the opportunity to match the criteria to the following possible response options:
- Scrum much worse
- Scrum worse
- Scrum about the same
- Scrum better
- Scrum much better
The aggregation of these results told a distinct and positive story about Scrum’s efficacy as perceived by the collective.
Costs versus benefits
Now irrespective of all of the warm, fuzzy results you hopefully obtain from your surveys, these shouldn’t be the only indicator of whether Scrum is going to be a raging success across the entire organization. Remember that unlike an isolated pilot project, Scrum in the broader organizational context doesn’t operate in a vacuum. Transitioning from a controlled trial to a broad adoption introduces a range of new and often significant impediments that must now be considered. These could include: co‐locating cross‐functional teams, modifying incentive schemes, adjusting the office layout as well as overhauling sign-off procedures and client contract structures – just to name a few.
The cold hard reality is that in some organizations, the initial Scrum rollout may only serve to identify the various environmental and cultural constraints that will impede the successful implementation of Scrum. The organization might simply not be ready, willing or able to remove the necessary constraints.
Are we getting better?
Back to more positive news and let’s say that Scrum is alive and thriving in your organization. It is only natural then to want to gauge whether you are stagnating or progressing.
To make this assessment, there needs to be a relative benchmark that is pegged to either your past performance, or your performance relative to others (particularly competitors).
Luckily for all of us, Cohn and Rubin have taken much of the pain away through the creation of the Comparative Agility website. They explain that:
Comparative Agility
In Comparative Agility, we assume agile teams and organizations strive always to be better than their competition and their past selves. As such, there is no holy grail or “perfect ten” score to be achieved. In fact, there’s no predefined best‐in‐class or “Agile Maturity Level 5” to be achieved. Rather, Comparative Agility assessments present the results of a set of survey responses in comparison to some other set of responses. For example, using Comparative Agility it is possible to compare a team, project or organization to:
- The total set of collected responses;
- Responses from organizations in the same industry
- Responses from similar types of projects (such as commercial software, websites, and so on); or
- Responses from projects with similar lengths of experience at becoming agile.
The 75 questions of a Comparative Agility assessment are organized into seven dimensions and thirty-two characteristics. The seven dimensions represent broad classifications of changes to be expected of a team or organization as it becomes more agile. The seven dimensions are:
- Teamwork
- Requirements
- Planning
- Technical practices
- Quality
- Culture
- Knowledge creation
Each dimension is made up of three to six characteristics and a set of questions is asked to assess a team’s score on each characteristic.
The survey is user friendly and offers some fascinating insight into relative progress.
I’m positive about tools like this written and facilitated by genuine experts; however, I do have a warning for you; if comparing to others is the only benchmark that you use then in essence, you’re letting others dictate your progress and that can actually lead to stagnation. For example, if you take the survey and realize that you’re doing particularly well in relation to other organizations, you may be misled by a false sense of security. Firstly, not all of your competitors may have taken the survey, but worse than that, you may feel that you are in the luxurious position of being able to enjoy your time at the top and take your foot of the continuous improvement pedal and that is never a good idea…
Keep it simple
If you really want to keep things simple, just ask three questions of yourself:
- Are your clients happier – are they remaining loyal and buying more products?
- Are your team members happier – are they abandoning ship or are they smiling more?
- Are your stakeholders happier – are they more relaxed and letting the team do its job rather than micro-managing?
In addition, the metric that should really interest you is the rate of Scrum adoption growth. Is Scrum delivering such great results that it is now beginning to spread across the broader organization, even crossing the chasm into other non-software related departments? For me, that is the ultimate success!
Spread the good word
In Australia, it is widely accepted that culturally, we’re not very good at celebrating success – this trait even has a label believe it or not – ‘tall-poppy syndrome’. Being humble and modest can at times be admirable, however, when introducing change, celebrating success is critical to help maintain momentum, generate excitement and encourage those who have been taking the risks. So, in this case, I say forget modesty and let the whole world know how well things are going! Conduct regular team meetings to discuss progress. Invite senior stakeholders to these sessions. Circulate emails with anecdotal stories to back up the statistics. Sing success from the rooftops! I don’t care how you do it but you should celebrate your continual improvement and let the team and the organization share in the glory!
If you liked this article, you can:
Subscribe to this RSS feed!
Find out more about Scrum adoption and progress metrics by taking one of our CSM training courses.
Thanks for a nice blog post! Interesting things indeed.
You were asking for examples of metrics. We have been using velocity (usually mature teams have stable velocity), number of open bugs (to indicate if technical debt is piling up) and the amount of scope change. I think this last metric rather well characterises how well a team has been able to give up the old ad-hoc adding of issues and learned to plan and follow that plan. It also measures how well the team is guarded from outside interference.
Customer satisfaction is a good measure, but depending on the nature of ones operations, it’s not easy to see the impact of your development actions on this very often.