Monday, May 20, 2013

Agile Metrics

Metrics play an important role in project management. They are the primary way to monitor and communicate the status and progress of a project to senior stakeholders. I already mentioned several metrics in previous blogs, but I’d like to sum them up all together.

Velocity
The first metric, and one that was defined in Agile and has no counterpart in traditional project management, is the velocity. This is defined as the number of story points per sprint that can be delivered by the Team according to the Definition of Done. The velocity is important because it will tell you when all scope on the backlog will be done, it will tell you when you run out of scope. If the velocity drops from one sprint to the next, there should be an explanation for that. It may be that some team members fell sick. Maybe there where a few national holidays. Maybe the user stories that were put on the sprint backlog where more complicated than thought. There can be a wide variety of reasons why the velocity varies between one sprint and the next or why it deviates from the average so far. If those reasons are one-offs, you need to see if there is a way to make up for the loss to keep the project on track, or have the Product Owner come to accept the drop in scope. If the reasons are structural, you need to make the Product Owner and senior stakeholders aware that there is an issue and that expectations must be adjusted.

Burn Rate
The burn rate is the budgetary counterpart of the velocity. It is defined as the amount of man days that are spend during a single sprint, by everybody who books on the project. If your project has several scrum teams, you may want to split out the burn rate for each of the teams. If there are people booking on the project who support the teams, like business analysts, product owners, anyone who is not part of a scrum team, then these costs should be evenly distributed among the teams as to get a burn rate per team that includes indeed all costs that are made to have that team deliver software. Where the velocity will tell you when you run out of scope, the burn rate will tell you when you will run out of budget. If you have a fixed deadline, then the velocity tells you what will be delivered by that deadline and the burn rate will tell what you will have spent.


Defect Detection Rate 

The defect detection rate is the amount of defects detected per sprint. Assuming that developers produce defects at a more or less constant rate, it is correlated with the velocity; the more story points are delivered, the more defects should be found and fixed as well. Teams tend to be pretty consistent in the quality of the software they deliver, so a drop in velocity combined with a rise in the defect detection rate should trigger the alarm. Something’s cooking and you need to find out what it is. My personal opinion is that a lower defect detection rate isn’t necessarily better than a higher one. A defect more found in one of the development and test environments is a defect less that makes it into production. From that perspective, you could support the statement, the more defects the better.

Defect Closure Rate 

This is the amount of defects fixed and closed per sprint. It should be equal to the defect detection rate. If it’s not, the amount of open defects will rise as you move along with the project, leaving the largest part of the bug fixing for the end of the project. This brings me to the last metric.

Gap Between Total and Closed Defects 

This is the difference between the total amount of defects and the amount of closed defects at any one time. This number should be as low as possible. A low number indicates that the quality of the delivered software so far is good. That implies that there will be few if any surprises once UAT and release preparation starts. And that in turn implies that the velocity and burn rate you have measured are indeed reliable indicators to forecast the remainder of your project. I consider this the most important metric of all, for if it’s low, it means I can indeed rely on the other indicators.

A healthy project has a stable velocity and burn rate, combined with a stable and sufficiently high defect detection rate and a low gap between total and closed defects. The velocity and burn rate will ideally indicate that you will run out of scope before you run out of budget, and that you run out of both before the requested delivery date.


It is not possible to give here absolute numbers for any of these metrics that would indicate for a random project whether or not it is in good shape. For instance you can’t say, a project with x number of developers and y days per sprint should produce no more than z number of defects per sprint. Such statements are nonsensical. The actual values of these metrics will depend on the technology you build your systems on, the developers and testers you have, the tools and practices they use, the existing technical debt if there is any, the functional and business context the project is executed in and many, many factors more. What matters is that you determine the actual numbers that result from the execution of your project given its current context and that you know how to interpret them, so you can act accordingly.

No comments:

Post a Comment