Software Project Estimation: Case Study of Estimation Impacts on Deliverables

Rich Ellison, The QA People

Estimates are the cornerstone of completion for any project and always a challenging item on a project to address. One of the main reasons it is so difficult to work with estimates is that they are often put together under duress of a predefined timeline. This paper reviews a case study as an opportunity to define a clear approach for estimations on software development projects. The method discussed is the combination of two approaches previously put forward. The first is a straight-ahead approach of using specific estimations for specific goals. The second approach modifies the first in regards to the information you have on the activities.

To give a little background on my experience with estimation, I would like to use my first project as my case study. When I moved from database management into software testing my first project was one of those laboring projects as it relates to estimation. We were late on development, the timeline was getting squeezed and we were starting to run out of room to complete. On one of the final calls regarding this with senior management things finally came to a head. "Ok, I just want to cut to the chase. When will it be done? I need to know if we have to move the delivery date so I can prep for the impacts to other projects." A stern annoyed gentleman speaks into the conference call for the project team. It was the closest he could get to yell. The truth was quite simple: we did not know. But when you are talking to VP and the Presidents of a company you do not have the right to not have an answer. The project manager spoke calm and clearly, "We estimated the additional effort and it looks like we will be delayed by three additional days which would put us into the freeze period. We will still be able to make the release date but we will have to reduce the timeline into the testing window by a few days. We confirmed with QA that it is possible and would not impact anything overall. "The VPs and the Presidents were calmed. They asked a few follow up questions but heard what they wanted to hear: that their new functionality would make it into the window they had set and would obviously not have any issues and would make an insane amount of money for them. The clients would be happy. Senior Management would be happy. It would be a daily parade of "thank you". Right after the phone hung up the project manager looked at the team and said that the timeline has to be met. The developer spoke up "I am sorry I need to let you know (a few things). I told you I had no confidence in those dates. You asked when do you think this will be done and I said: I don't know maybe 3 days maybe 5".

I did not know anything as a low lever tester at that point. I take that back I knew 4 things.

1. It was not my fault because test had not even started.
2. We did not have a clue what we're doing and how we we'd do it
3. We did not know when we would be able to complete it
4. It would never be on time.

Number one showed how young I was in the field, because when problems start rolling downhill on a project, it does not matter if it is your fault, as it quickly becomes a shared pain. More importantly number four was actually wrong as well. We did make it on time. Three additional developers were added. Everyone worked nights and weekends. The development was pushed back to one week prior to release. Testing would be at a high level overview of the enterprise application with a few days focused on the new functionality. Development and testing was going blind since the business analyst wrote the requirements after it was coded so we would "have a detailed list for future reference". On the day of the release, the build hit the floor DOI (dead on installation). It crashed a few things and caused a series of headaches. The final version released was not the same in production and development since developers were asked to add one final last minute page to introduce the functionality. Developers worked on it that night to add the new functionality and it killed the site for 4 hours. People were mad and it was everyone's fault. We worked for 4 days to clean up our mess and scraped the functionality. Right after that management had someone offer the team a three-day intensive training on software development life cycle. We initially took it as a way of punishing us. Since then, I have always had a desire to know more about estimates. I have always tried to keep an ear open to new studies and approaches. Looking over the project from beginning to end, it is obvious to see where we went wrong.

It was obvious our estimates were wrong even though we did meet our timeline. We had more important warning signs aside from the scheduled that would have tipped us off. We estimated against one section of work and added additional work to that. Since we were having issues with our initial scope, the project should have been moving in the other direction and looking at reducing scope instead of increasing scope. The team also did not know the scope of the new effort that was being added which is a huge red flag as well. In short we had a project that would never end and the dates were slipping day by day. The cost was increasing and people were overworked. I would later learn that each of these alone are telltale signs of poor estimations and would raise a red flag by themselves.

The first issue was that this project had one estimate created. Initial estimates are expected to have a greater degree of variability and should be become more precise as the project moves forward as shown below in Steve McConnell's book Software Project Survival Guide.

The process that should have been followed was a rough estimate which should be 25% to 50% off the final estimate or as a co worker would define it, "a wild guess with no backing or reason". I agree with part of that. It is a wild guess, but it does have some reason behind it. The estimates should be completed at least two times. Once at the beginning of the project and one later on the project. If you do a detailed planning for all of the requirements as your initial estimate, you will not be helping yourself at all. Detailed requirements are not set at the onset of a project so you will find yourself accounting for a bunch of things you will never build and not accounting for a bunch of things you will have to build.

The second issue we had was that the project team did not know how the estimates were created. When someone comes up to ask for a project estimate for any new functionality it is good to know the types of estimates that are available for you. What is really an estimation? An estimation is "a prediction or projection of a quantity based on experience and /or information available at the time. Estimates are the least accurate model for an outcome because it is supposed to be incorrect. Below are a list of the types of estimations and some additional details on some of the approaches.

Estimation approach	Category	Examples of support of implementation
Analogy-based	Formal estimation model	ANGEL, Weighed Micro Function Points
WBS-based bottom-up	Expert estimation	Project management software, company specific activity template
Parametric models	Formal estimation model	COCOMO, SLIM, SEER-SEM, True Planning for Software
Size-based estimation	Formal estimation model	Function Point Analysis, Use Case Analysis, Software Size Units, Story Points
Group Estimation	Expert estimation	Planning poker, Wideband Delphi
Mechanical combination	Combination-based estimation	Average of an analogy-based and a Work Breakdown Structure-based effort estimation
Judgmental combination	Combination-based estimation	Expert judgement based on estimates from parametric model and group estimation

Analogy Based Estimating

Analogy based estimation have been heavily studied with varied results. One study compares various types of analogy-based software effort estimation with each other that showed that people are better than tools at selecting analogues for the data set used in this study. Using a tool along with that data proved even more accurate. Other studies have shown the use of classifications has also increased the accuracy. Other estimates have shown that your effective rate requires a lot of additional data for computation. Tools like Angel are used for the collection and storage to identify for new projects.

Bottom Up and Top Down Estimates

Bottom up is a resource driven approach that takes the project and breaks it into groups and then into smaller items until you get to the task level where you the estimate the actual effort. Top-down (or parametric) estimating typically relies on actual costs of past projects, and knowledge of the specific parameters of those projects. Using the previous example, reviewing previous bridge projects to determine that there are correlations between the cost of the bridge and key factors like size and the primary material used gives the estimator valuable data. Top Down will be a high level estimate for the early part of the project while bottom up estimates can be used later as you have more detail to offer the detailed estimate. The practical reality is that both approaches are likely to be used by most companies and on most projects. top-down may be the only approach possible for a conceptual estimate for an early-stage project. And even later-stage estimates might involve a mix of items, some estimated top-down and others bottom-up.

Size Based Estimates

A function point is a measurement to show the number of business functions a system provides to the user. Function points are used to define functional size measurement (FSM). The cost is used from previous projects. It is based on a group of "functions" with 2 data points and 5 functions (2 data and 5 transactions), The are counted and weighted by low medium and high as shown below.

What estimate type should we have used? There is a substantial dependency on the organization and the data and the tools that are available. One approach that seems simple but effective is to look at the phase as the initial indicator and then at the activity type for addition guidance to define the estimation approach. There are many ways to break the activities down into groups. I am using the approach which is I was initially presented with stable activities, dependent activities, uncertain activities and unknown activities. First I will use the approach Kathleen Peters put forward about the estimations required.

Estimation for Initial Scope

In the initiating phase you will need to determine the basic size of the project. To determine the project size you can use one of two approaches. You can use analogous estimate or using a size based model such as function point analysis. For the analogy it is important to not just look for a similar project. You will instead look for a project component that is similar and perform the same tasks for each of the components of the project and add up the total. This issue with this approach sometimes can be that companies do not collect this data on projects. Function points or other size-based models will use an algorithm defined by the system functions and give you the size. Your output for this will be based on the input and algorithm that is used. Some of the studies have shown data that points to analogy based estimates to be more accurate (in some cases) but even the conclusion from the study determined the results were inconclusive.

Estimation for Effort

Once the cost is defined you can start to look at the effort that will use your analogy-based estimates or your parametric based estimation tools to use the statistical relationship to define the duration. One tool is COCOMO that has three classes of project that are created by size and experience and defines the effort, time and resources.

Estimation for Duration

Using the information above you will create your work breakdown structure to define you actual schedule for each task using a bottom-up method. Where possible, use the organizational data do define the resources required to complete the activities. There are three types of activities that will need to be estimated for the schedule and depending on the types of activity you are blocking time for, your approach may have to change. Using a simplified approach the activities will be broken into stable, dependent and uncertain.

Reviewing Activity Types as A Dependency for Your Estimates

Using Dick Billows approach from 4PM I have listed the activities grouped by the information available in each activities to define how to modify your estimates to give a more accurate results.

Stable Activities are just as they sound. The tasks are fully defined items that do not have much volatility. This lends themselves to the Formal Estimation Approaches listed above. To define if you are going to use analogous, parametric or size based models review the information you have available as well as the resources to help estimate to allow you to select the correct approach.

Dependent Activities are unlike the stable activities in the fact that some factor does not allow you to have all of the information to have a complete picture of the duration of the tasks. For example, you have a complex technical approach which will be defined later in the project but you need to estimate for not only that technical work but also the subsequent testing work that would follow. For these tasks the method would use an analogous estimate.

Uncertain activities occur as a project gets more complex and you will have more components that are unknown. You will have to go back to the scope estimate and work through the refined estimation process to get to the details for these activities. As mentioned in Dependent and Uncertain Activities, complexities are a dependency that should be reviewed and taken into account for each project. The use of a complexity factor can assist with the analogous estimations.

Looking back now, the project I mentioned could have been helped with proper estimation to avoid a long list of headaches that followed. It still had the issue of scope but it could have gone across the finish line easier with that estimation knowledge. In retrospect understanding that estimates are created at various points and are monitored through the project would have been a good starting point. Also understanding that estimates for work is not a random information that the project team blindly receives would have tipped us of that the approach would have been incorrect. Most of our issues would have been solved knowing that you have various approaches that can serve purposes for each phase and can be modified by looking at the information you have on the activities required.

References

Bundschuh M. - Function Point Prognosis - FESMA 98 Procs, May, 1998

Catherwood B., Sood M., AMS - Continued Experiences Measuring OO System Size - ESCOM 97 Procs - May, 1997

Fetcke T., Abran A., Nguyen T.H. - Mapping the OO -http://maxwideman.com/issacons3/iac1330/sld008.htm

A Simulation Tool for Efficient Analogy Based Cost Estimation Empirical Software Engineering March 2000, Volume 5, Issue 1, pp 35-68

An Empirical Study of Analogy-based Software Effort Estimation Empirical Software Engineering Volume 4, Issue 2, pp 135-158

Proceedings - International Conference on Software Engineering April 1996 with 678 Reads

IEEE Xplore Conference: Software Engineering, 1996., Proceedings of the 18th International Conference IS YOUR ESTIMATION TOOL WORKING

Conference Paper (PDF Available) Is Your Estimation Working, Kolinger and associates

Adele Sommers, 12 Tips for Accurate Project Estimating

Estimation of project time and costs, www.csun.edu

Peters, K., Estimation Basics, Washington.edu

---

Related Project Estimating Articles and Resources

#NoEstimates - Alternative to Estimate-Driven Software Development

Estimating With Use Case Points

Fingers in the air: a Gentle Introduction to Software Estimation

Project Management Planet

---

Click here to view the complete list of archived articles

This article was originally published in the Fall 2016 issue of Methods & Tools