Dominique Stender A blog about my thoughts and experiences in Information Technology

Estimation techniques compared

I was inspired to write this article by several threads in forums covering agile methodologies. The basic discussion was whether to estimate based on Story Points or time. To me - and most people participating in those discussions - there is no silver bullet and you will have to find out what works best for you yourself. In this article I'd like to muse about the various estimation techniques that I'm familiar with as well as the pros and cons that I ran into with each of them.

I will compare PERT and COCOMO, both being traditional techniques with a long history and Planning Poker, probably the most prevalent agile estimation technique.

All techniques have their pros and cons and it will be up to you to determine which technique will work best in your environment.

My comparison is purely based on experience, which is broader for PERT and more narrow for COCOMO and to some extend Planning Poker. Your mileage may vary. This article does not intend to be the definite guide on estimation techniques.

That being said, let's start shall we?

PERT

The PERT estimation is the standard estimation technique at my company. It is time-based and probability-based. The basic formula is to estimate an optimistic time (Topt), realistic time (Treal) and pessimistic time (Tpess) for an activity and generate a mean value (Tm) based on the formula Tm = (Topt + 4 * Treal + Tpess) / 6.

The realistic time Treal is weighted four times higher than the optimum time and worst case time, the sum of all three is divided by six.

For example if a task might complete in two days if everything runs fine, in three days under normal conditions and in five days if we run into complications the PERT value would be (2 + 4*3 + 5) / 6 ~= 3.2 days.

If you estimate all tasks of a project with PERT you can generate the variance and get a buffer value out of it which can be pretty handy. Don't trust that buffer blindly though.

One of the pros for me is that PERT acknowledges the fact that humans are particularly bad in estimating a task down to a single number. The thought of optimistic, realistic and pessimistic cases alone works wonders in discussions. The formula usually does a decent job to come up with a mean value.

Also a pro for PERT is that it is reasonably fast. Estimating a 3 month project on a level of abstraction needed for an offer usually takes no longer than a couple of hours.

My biggest con against PERT is that it is easy to misuse it. What I see frequently is that a discussion starts whether or not a particular task has a risk. Often that leads to optimum and realistic estimation being the same, let's say 2-2-5 instead of 2-3-5 as shown above. Or in a similar manner, realistic and pessimistic estimation are the same: 2-3-3.

Instead of the above mentioned 3.2 days for 2-3-5 we get 2.5 days for 2-2-3 and 2.8 days for 2-3-3. In a nutshell that ruins your buffer. You do not want that.

Another more general issue I have is that PERT is time-based but I'll get to that in a bit.

COCOMO

For me COCOMO is the rocket science approach for estimations. It is complexity-based and rooted on calibration through historic data. The algorithm behind COCOMO will get fine tuned over time, while your "database" of historic data grows.

The key figure in COCOMO is the lines of code (LOC or SLOC), more frequently thousand lines of code (KSLOC). You need to know - or in the first project, estimate - how long it takes to produce 1000 lines of functional code. Analysis, design, coding, testing, ... included.

Then you need to do an estimation of how many lines of code will have to be written or changed for the new project.

A set of factors or "cost drivers" such as project complexity, required quality, team experience etc. is applied to the estimations and after putting all this into a formula you get the estimated amount of time.

By far my biggest issue with COCOMO is that it is slow and too complex. You can't get decent estimation quality with COCOMO in a short manner of time. Even intermediate COCOMO takes into account no less than fifteen cost drivers, all of which affect each and every task. The bottomline is that you need a tool for estimating.

Disclaimer: There is a variant of COCOMO that is faster than traditional COCOMO. Personally I don't find that to be very exact, though and hence not very useful.

The second con against COCOMO is that it is fragile. Change your team constellation from one team to another and your historic data is useless. The same applies more or less for all other cost drivers.

I like the idea that COCOMO gets calibrated with time through your historic estimations. But in the reality of changing teams, hardly comparable projects and time pressure that leads to historic data not being available in a clean form frankly I never saw that work. Granted, that is not a failure of COCOMO but rather the methodology not working for us.

What I generally like in COCOMO is that it is complexity-based rather than time-based.

Planning Poker

Planning poker is an agile estimation technique. As the name implies it is based on playing cards (my .pdf for download) . The estimations in planning poker are relative, complexity-based and consensus-oriented.

Each planning poker card will show a number: "0.5-1-2-3-5-8-13-20-40-100-?-infinity" are a common set but there are other variations. The card "?" indicates that the estimating person can't estimate the task and "infinity" translates into "really, really huge" and can be seen as a sign that the task should be broken down into several smaller tasks.

The estimation works as follows: Each team member has a set of cards on his own. A task is explained, usually by the product owner / customer. A discussion takes place to clarify doubts from the team. After the task is clear, each team member picks a card out of his stack secretly and in unison they show the cards to each other.

If the numbers on each team members' card are reasonably close together (i.e. 8-13-8-8-5-13) a rough mean value is "calculated" (probably 8 in this case) and the next task comes up for estimation.

In case the numbers vary wildly the team members with the lowest and highest numbers explain why they picked their number. This is to reveal misunderstandings and generate a broader understanding for all team members. After that a second estimation round is performed. The process repeats until all numbers are reasonably close.

By now you may ask what the numbers are indicating? They do not indicate time. The card "5" does not mean "five days" or "five hours". The cards indicate complexity. There is no unit for the numbers, they are commonly referred to as story points or points for short.

What points show is that a task with a complexity of 2 is twice as complex as a task with 1 point and roughly half as complex as a 5 point task.

When estimating with planning poker it is absolutely mandatory to understand that each tasks complexity is only relative to another task. So what you do is you pick a comparable task that has already been developed (from your last project or the last sprint in the current project) and assign a medium sized number to it, for example "eight". Now for the current bunch of tasks you ask your team to compare the new task to that one and pick a card from their stack based on its relative size.

What you get is a consensus-based estimation how big or small a current task is related to one that is well known and understood. Since a task with complexity "eight" is almost twice as big as complexity "five", and you know how much time "eight" took to plan, develop, document and test, you have a rough idea how long the new task with complexity "five" will take.

My biggest pro for Planning Poker is that it will enable your team to estimate their own work. Usually this will take a few iterations as they need to get to know their own pitfalls but the long term benefits clearly outweigh the initial pains. Once the team knows how to estimate, you will not find anyone outside the team who can estimate the teams own work.

Another big pro for Planning Poker is that it is by far the fastest approach and that it is fun. Calculating numbers for hours and hours is boring, planning poker is more of a focused discussion.

One negative aspect of Planning Poker is that it is somewhat fragile. If your team size changes often, getting good estimations out of Pllanning Poker might be hard. But in agile the team constellation should stay constant even more than in traditional methodologies.

Bottomline

That's it, the three estimation techniques I'm most familiar with.

If you work in a traditional environment, give PERT a try. Use COCOMO if you can afford the overhead and if you have comparable projects in a very stable environment.

In any case I urge you to try out Planning Poker. While the methodology itself is agile, I don't see why Planning Poker should not work for you, even if you're in a traditional waterfall of SDLC environment. It will give your team a much better understanding of the upcoming work and enable it to participate in a project at a much earlier stage than usual. Through that, the team will feel involved and motivation will go up.