Scope vs. Effort

When estimating and executing a project, it is important to have a good view on effort and scope. In order to do so you need to have a metric that you can use to measure these. For effort it is obvious, man days, man hours, man weeks will be your unit. Often, the same units are used for scope as well. In Earned Value Analysis you have something called a budgeted man-day (or budgeted dollar) which is used to define both scope as estimated effort. Your actual man-day’s or dollars will then tell you how much you really had to spend in order to achieve a budgeted manday or dollar. In this setup, estimated effort and scope are treated as one. Personally I find that confusing, since the same unit of measurement is used for two entities. Another drawback is that you are expected to come up with pretty precise man-day estimates even for the smallest items. That means a lot of estimating to be done up front, and the relevance and accuracy of man-day estimates for small items a long time before they are implemented is questionable.

Of course this one of the reasons why the Agile community came up with the concept of story points. I must admit I needed a while before I learned to appreciate the story point. Initially I found it too abstract, and I had difficulty in relating it to budget. My reasoning went as follows: It is nice to have a project estimate in story points, but you can’t go to your project sponsor or CFO and tell him that this project is going to cost 750 story points. Senior stakeholders need a dollar figure in order for them to calculate a ROI on the project to see if it makes sense. So at the end of the day you still need to convert your story point estimate to man-days and dollars, and if that is the case, then why bother with story point estimating at all? It just seemed making estimating unnecessarily complex.  So up until a few years ago, I did all my estimates in man-days and use Earned Value Analysis to track progress.

But more recently I realized that the good thing about story points is that it can be used as a measure of scope, independent of effort. The two are related of course through the concept of velocity (story points per sprint), but they are separate entities that have their own measurement unit and can be estimated separately. This means that a team can have discussions on the contents of a user story or epic without thinking about the effort. They can decide on whether user story A is larger or smaller than user story B and as a result they assign more or less story points. There are several ways of doing this (planning poker and comparing with a baseline user story, or sorting the user stories by size and give the smallest one a value of 1 and then go up the Fibonacci series). Once all user stories and epics are estimated in story points at the beginning of the project, and many functional and technical issues have been addressed during the estimation process, putting a man day figure on the project as a whole becomes a lot easier. I explicitly consider the story point estimate as a measure of scope, because I can then also track it separately from effort. As experience learns, there are cheap story points and expensive story points. Over the whole project you will average out a certain velocity, but some user stories that have a low estimate in story points will turn out to be expensive in actual effort, and some large user stories will come in pretty cheap. That’s just a consequence of not knowing everything up front and unforeseen events that impact the actual effort. In other words, you will tend to overestimate some user stories and underestimate some others, and you will encounter team members falling sick or leaving or on the contrary maybe feeling more productive than usually. This means that the relationship between scope size and actual effort is a statistical one, not a deterministic one. On average you will achieve a certain velocity but for an individual user story you may either achieve that velocity, or go faster or slower. That will depend on the things you didn’t know when estimating the story and circumstances you encounter while you are implementing it, both of which you don’t have under control at the start of the project.

In Earned Value Analysis, this would come up as some budgeted man-days being cheaper in actual man-days than other budgeted man-days. Again, I just find that confusing. I prefer to work in story points nowadays. If we add a new user story, we only consider its size relative to the ones we already have on the backlog. If we are not entirely sure, we assign more points to incorporate the risk. We don’t bother with getting a precise man-day estimate. After adding a new story, our scope has increased with some points and our project burn-up chart will tell us if the total scope is still feasible within the allocated budget and schedule or that we have to remove something.

All fine and well, but as I said previously, CFO’s don’t care about story points when the time is there to assess projects on their potential ROI. So how do we go from story points to man-days and dollars? If you have an existing team with an established velocity this is straightforward. Just divide the amount of story points they estimated through their velocity and you end up with the amount of sprints. From there it is easy to get to man-days and then via their daily rates to dollars. If you have a newly composed team with still unknown velocity, you can approach this in several ways.

The first is to let the team establish  an initial value for the velocity. Let them take a few user stories again, of different sizes and try to estimate these in man-days. Then compare the results to the story points of those user stories. Continue with this until the man-day estimates for these stories are more or less in line with the story point estimates i.e. a 1 point story results in an X days estimate, a 5 point story results in more or less a 5X days estimate. That should give you an indication of story points per man-day. That you can translate to an initial velocity and total project man-days and dollars.

Another approach is from a resourcing perspective. You can ask the team the question, how much time would this entire project take us, and which team composition would we need for it, based on the information we learned estimating the initial user stories and epics? The result will be a man-day estimate for the entire project, and you can work your way back to an initial velocity.

Or you can let them plan the first sprint based on what they think is realistic and take that as your initial velocity. Again just divide the amount of story points through this initial velocity and from there go to man-days and dollars.

Bug tracking and monitoring

Developers make bugs. There’s nothing you can do about it. So in order to deliver a high quality project the key is to find them and fix them. That means you need a view on how much bugs have been found vs. how much are potentially in the already developed code, and you need a view on how much of these have been solved. If you don’t have a clear view on either of these parameters at all times during your project, your project has an unknown quality. That means that your schedule and budget are becoming unreliable, since you don’t know how much time and money you still need to spend on bug fixing.



A useful indication of how much bugs you have found vs. how much might be in there is the amount of defects logged per developer man day spend. If you have historical bug and time tracking data on projects on the same code base performed by more or less the same people, you may be able to determine such an amount and compare it to the ratio of your current project. In the projects I’ve done so far, this number is between 0.5 and 3 bugs per developer man day spend. I don’t make a distinction between creating the bug and fixing it, I just take the total man days spend by developers, irrespective of their activities. I also don’t distinguish between types of bugs, so even if the bug cannot be reproduced or if it’s due to a misunderstanding of the functionality, I still count them. For instance, if you have seemingly no trouble to get your user stories accepted, but your bug detection rate is lower than what you would expect, that may be a sign you need to dig a little deeper to find out why.  Are the test scenarios thorough enough? Have developers improved their unit test code coverage leading to less defects being deployed in the test environment? It is important to know this because you may have a false perception everything is going all right during the project, only to be confronted with many post release issues that should have been avoided. Similarly, if the bug detection rate is higher than you anticipate, it is important to know why. Your developers may have less experience. Your user stories may be subject insufficiently elaborated. If you have more bugs than anticipated, you need to spend more time fixing and verifying them, so this may impact the amount of scope you can deliver within the set schedule and budget.

You also need to know how many bugs are fixed and verified in comparison with the total amount of bugs. In order to be confident in the quality of the already implemented code, these to numbers should be as close together as possible. If you have many open bugs, that means that you need to spend time and money on fixing them, and that the functionality you have delivered so far is of inferior quality. The way to handle this is to give bug fixing priority over all other tasks from the start of the project right until the end. It makes sense from a developer standpoint as well. A bug is fixed fastest if the coding he has done is still fresh in his memory. If he needs to address a bug a few weeks or even months after he created it, he will need some time figure out and remember what he had done exactly. If you have a well defined Definition of Done that you apply consistently to all you user stories, this comes in a natural way, since an open bug means a user story not done, and a corresponding drop in velocity. If you do this, and you log your total and open amounts of bugs at the end of each sprint, you should be able to pull a graph more or less like the one below:

You can see that during this project, the red line closely follows the blue line. Only at the end of sprint 16 there is a somewhat larger gap between the two. That means that at that point some action had to be taken to close the gap again. This can be overtime, or taking on less user stories in order to focus on bug fixing. In any case, it was dealt with because in the following sprints you see the gap closing again.

In a more traditional waterfall like approach, this graph would look somewhat like the one below:

You see here that the defect detection rate goes up very fast during the testing phase, and levels off towards the end of the project. Up until the leveling off starts, you basically have no idea how much bugs are still in the code, and you may still be in for negative surprises. Also, you don’t have actually reliable working software until just before the project ends. That means it is difficult to get any functional feedback from end users, since they need to work their way through the bugs first when assessing the application. Finally, the graph indicates that developers spend a good chunk of their time during the last stage of the project fixing bugs. A seemingly never ending flood of bugs can be demoralizing for your team, since they feel they have no control over it. Even very good developers may resign over situations like this, at the time when you actually need them most.

Project baselining and tracking

As a project manager, independent of methodology, you need to keep track of four things: Scope, schedule, budget, and quality. Nothing new here. In Agile, the favorite method (or at least my favorite method) of keeping tabs on your project is through the burn-up chart. This chart covers scope, schedule and budget. Tracking quality will be the topic of another blog post.

You start out with a certain amount of scope for your project, that you can express in story points, ideal man days, dress sizes or whatever metric you choose, as long as it is numeric and everybody involved in the project understands and agrees on its meaning (There will be future blog posts discussing the various options of measuring scope and their implications, as well as estimating the initial scope of a project). Please note that scope is treated as a variable through the project, we keep budget and schedule fixed.

The schedule follows from the speed at which you implement scope; this is of course known as the velocity, the amount of story points implemented per sprint. In order to baseline your project, you must have some idea of the velocity up front. If your team has been working together in the same composition in the previous project, they will have a pretty good idea of their velocity. If the team works together for the first time, you need to discuss and agree with them on a velocity that they feel can realistically be achieved. If possible, make corrections for known holiday periods or other absences. The alternative is of course that you use the first few sprints to determine this velocity, but then you cannot give a baseline schedule up front. This may or may not be an issue depending on your business processes regarding project funding, approval and kick-off.

You will also have a fixed budget within which your team must deliver the scope. This budget can be expressed in real man days or a currency. The translation from scope to budget must also be agreed upon with your team; at least as far as man days are concerned. The actual dollar amounts will depend on the rates and salaries of the team members, and also invoicing and payment cycles are not likely to match your sprints. Tracking the budget in monetary terms is therefore a little more complicated, but there are no fundamental reasons against it. Since you have already established a baseline schedule based on your expected team velocity, you can also produce a baseline cost progression based on expected team member availabilities. I will refer to the speed with which you spend budget, i.e. the amount of man days or dollars per sprint, as the burn rate.

So now we have four metrics: baseline scope to be delivered, baseline delivery schedule based on expected team velocity, a budget and a baseline cost progression based on expected burn rate. You can plot these in a graph versus time, in units of fixed length sprints, see the figure below.

In this graph you see that up until sprint 10 the expected velocity is the highest. That’s because one of the teams is expected to leave the project after that. From Sprint 20 on, only a minimal team will be taking care of post-release work. The baseline cost progression follows the same pattern. Scope is expressed in story points, budget in man days.

In order to make this a burn-up chart, you need to update the amount of story points delivered (as per your Definition of Done) and the amount of man days spend after each Sprint Review. After several sprints, the result will look something like this:

You see here that the actual scope delivery is above the baseline, and that the actual cost progression is slightly under the baseline, so this project is going good. You also note that the blue project scope line has upward bumps at sprint 5 and 10. At this point re-estimations and new user stories resulted in additional scope. According to the scope trend line, this project will finish by sprint 15, and according to the cost trend line, around 1800 md will have been spend at that point. Of course the trend line does not yet take into account the reduced staffing that will come into play after sprint 10, making these projections for cost and schedule too optimistic in this particular case. However so far this project has performed slightly under budget and it is ahead of schedule since more scope has been delivered than projected.

A different outlook would be the one below:

This project is running behind schedule; less scope has been delivered than planned, and at the same time more budget has been spend. At the same time, additional scope has been added to boot. The scope trend line indicates that all current scope will be delivered by sprint 22 (but only if we keep all teams on board, so no reduction in velocity), and the cost trend line indicates more than 3000 man days spend at that point, a 50% overrun. This project is headed for disaster, and it’s time for immediate remediating action. And that action would be to start dropping non-essential scope items so that the blue project scope line goes down again, as well as tackling any impediments identified by the team as a reason why they need to spend extra time.