Estimating Part II

So what is an estimate really?  Maybe a stupid question but I think it is important that all parties involved, from developers and testers to the most senior stakeholders agree on the meaning of the estimates provided, and that includes agreement on its definition. So my definition for an estimate in the context of Agile and Scrum is the following: An estimate for a deliverable is a range or set of possible values for which that deliverable could be delivered according to the previously agreed upon definition of done, with a corresponding probability for each of the values in the range. The values can be expressed in story points, man days or dollars.

So basically we have a range or set of values with their corresponding probabilities. In statistics, that is the definition of a probability distribution. Assuming that you arrive at your estimates using planning poker, I am going to treat an estimate as a set of discrete values with their corresponding probabilities here.

When it comes to estimates, we want accuracy, precision and confidence. Accuracy is difficult to predict beforehand, after all, it is just an estimate. But there are some assumptions that if true lead to more accurate estimates and when false will lead to inaccurate estimates. These are:

1.  The user stories and epics have are independent, i.e. they can be implemented and tested and have value independent of the other ones

2.  The team performing the estimates is overall knowledgeable and experienced enough in the technology, business domain and estimating techniques. That does not mean they all need to be experts in all 3 areas, but working with a team consisting solely of recently graduated juniors is going to make this a much more difficult exercise than when all of your team members have more than 10 years working together on similar projects with similar technology.

On precision and confidence we can say something by doing some statistical analysis on the estimates provided by the team.

Say we have a team of 5 and we are playing planning poker to estimate 10 user stories and epics for a new project. Some user stories are very detailed and small and will lead to a lower estimate; other epics are only described on a high level and will be subject to change as the project moves along. These will receive higher estimates. The results of the estimating session are shown in the table below, together with mean estimates (μ) and standard deviations (σ) for the individual stories as well as for the project as a whole:

The last column is the 2-sigma value, which is approximately the limit of the 95% confidence interval if we assume that the estimates for a single story can be modeled as a normal distribution. That is of course for a single user story obviously not the case, but according to the central limit theorem, if we add up (or more precisely, convolute) the estimates for each of the user stories, then the probability distribution for the overall project will approach a normal distribution, the more so for more user stories, the less so for less user stories. In any case we may take the overall standard deviation of 46.58 story points as more valid for the entire project than the standard deviations of the individual user stories. So we have an estimate for the entire project of 87.2 ± 46.6 story points, with 95% confidence.

These 46.6 story points are 53.4 % of the mean so our precision with 95% confidence is ±53.4%. For user story A the 2-sigma value is 1.10, or 78.6% of the mean, so our precision with 95% confidence for this single user store is ±78.6%. So by convoluting estimates for several user stories, we get a more precise estimate for the overall project than for a single small user story. And that isn’t so strange if you think about it; if you estimate a single user story, you may over- or under-estimate it, and that’s that. But when estimating several user stories and epics, some will be overestimated and others will be underestimated and these will cancel each other out. The more user stories and epics you have the more opportunity there is for estimating errors to be cancelled out.

So what does this all mean? Well, a first conclusion is that you are more likely to come up with precise estimate for a large project than for a small one. This is a consequence of the central limit theorem. If I look back on my own projects, indeed I have had cost overruns between 1 – 12% on larger projects, but much larger ones on some of the small projects. All this for taking average estimates as my budget, plus a risk premium that didn’t even come close to the above mentioned 78% or 53%. OK, lesson learned.

A second conclusion is that you need to think about the confidence level that you could live with for your project estimates. If you want a 95% confidence, you need to submit a budget of μ + 2σ, if you want 99%, it will be μ + 3σ. If you can live with 70%, then μ + σ. What is this means is that you need to decide on the percentage of projects that have cost overruns you can live with. That number is your confidence interval.

Estimating

Estimating is a very important part of any project manager’s job. Wrong estimates lead to budget and schedule issues in your project. The best way to keep your schedule and budget under control is to be on top of your estimates, both before and during project execution. And you should be, as stakeholders and customers need to be able to rely on your estimates, for they have their own deadlines, targets, investment plans, marketing campaigns etc. that they need to plan as well. Delivering unreliable estimates and the corresponding unreliable schedules and budgets is simply unacceptable.

That being said, how do you come to reliable estimates for a project, if there is so much that is still unknown in the early phases, and so much to be learned and changed as the project is executed? When you start a project, make sure that there is a clear vision on what the project is supposed to deliver. This has to come from the product owner and project sponsor. Try to make sure that everybody involved in the project agrees on the vision; nothing can derail a project more than disagreement on this between senior stakeholders. Next, you’ll need an initial story map to start from. This has to come from the product owner. It should have all initial user stories and epics that would be necessary to implement the vision. The epics will be higher level and less precise, as the exact details will not yet be known. The most important user stories should be worked out pretty detailed, with acceptance criteria included enough to be able to start working on them in sprint 1. Other user stories will fall somewhere in between. Important is also that each user story and epic is as independent as possible from the other ones, i.e. it can stand on its own implementation and deployment wise, and it has business value independent of the rest. This is not always easy to achieve. Lastly, your story map must also includes enough nice to have’s. Since we consider scope our variable when we need to make decisions in order to keep the project on course, the story map, and later the backlog, must always contain some nice to have’s in case you need to drop something. If you don’t have these, you will be cornered immediately after a higher priority user story takes more budget and time than initially thought. And I think any project manager knows that this will happen.

Once you have this, you should think about which team would best be suited to implement this. In my experience, the best estimates come from teams that are also going to do the implementation. My experience with dedicated estimation groups who estimate a lot of projects that are then handed over to other project teams for execution is that they tend to miss a lot and produce unreliable estimates. But your experience may differ of course. I think it is a psychological issue; if you know you will be held accountable for the correctness of the estimate, and you also know you will have the power to take corrective action during the execution, you’re going to do a much more serious job. Another requirement for the team doing the estimation is that they have enough knowledge of the business domain and technology to come to sensible conclusions. That means you need at least a few experienced people on the team, who have done similar projects in the past.

So now you have a team and a story map. The next step is to schedule a few workshops with the team and the product owner to discuss the vision and the story map. This will lead to architectural and design discussions and decisions. It will also lead to changes to the story map, like new user stories, splitted user stories, rewritten user stories etc. Each time a user story is considered final, you can play planning poker to come to a story point estimate for it. Now of course you are not going to continue this until you have broken down all epics and user stories to 1 or 2 point stories. That’s just impossible. Usually 2 to 5 sessions spread over 1 or 2 weeks is enough the determine everything the product owner and the team know at this point in time, and the rest should become clear as the project moves along. So now you have a story map with some detailed user stories with small story point estimates, some less detailed user stories with larger story point estimates, and some epics with the largest story point estimates. All major architectural and design decisions should have been taken. The story map also contains some nice to have’s. The principal risks should also have been identified. At this point you have an estimate in story points for your entire project, which you can convert to a man day budget with 1 of the methods I described in my previous post. Or maybe you still have another method of doing that. Usually at this point I try to determine, together with the team, whether a risk premium is needed on top of the resulting man day estimate, and if yes, how high it should be.

When project execution starts, you will need to repeat this exercise regularly during backlog grooming meetings, so you can incorporate feedback and changes to the backlog. The backlog is of course a living entity, so some user stories will get dropped, new ones will appear, and other ones will need to be re-estimated based on what we learned. And of course it is vital that you keep track of any scope changes in your burn-up chart, so you can take immediate action when required.

OK, so you have your estimates, your budget and your schedule, and now your project sponsor or CFO says: That's too expensive to make a positive business case, make it cheaper. What to do? Well, asking your product owner which scope can be dropped, that's what you do. And again, make sure that not only the nice to haves are removed, because then you may run into trouble later in the project if one of your user stories comes in more expensive. If the project is like a pressure cooker, and the scope is what’s being cooked, then the bottom of your backlog is like its escape valve. If you don’t open it on time and let stuff out, the whole thing will blow up.

Another alternative that may be proposed is to look for cheaper resources. That’s fine, but I would at that point do the estimating exercise over again with the new team, because these cheaper resources may not have as much domain knowledge or experience, leading to higher man-day estimates. That would result in a different budget and schedule, than you would expect based on the daily rate alone.