GENERIC
DELPHI ESTIMATION PROCESS

@(#)delphi.htm 3.5 - 03/08/02
The Generic UWF Maintenance Process (GUMP) 1994, 1995, 1996 The University of West Florida. All rights reserved.

Permission is granted to reproduce and adapt this document provided credit is given to the University of West Florida. This documentation is provided "as is" and no warranty of fitness for any particular purpose is made or implied.

ABSTRACT


This document is the Generic Delphi Estimation Process document for the generic maintenance process for the University of West Florida. This document describes the estimation process, the activities required to be performed, and the associated responsibilities of each team member for estimating project duration for the generic maintenance process architecture. The method described herein employs the Delphi estimation technique.

TABLE OF CONTENTS

ABSTRACT
LIST OF FIGURES
LIST OF TABLES
1.0 INTRODUCTION
1.1 BACKGROUND INFORMATION
1.1.1 History of the Delphi Estimation Process
1.1.2 Employment of Delphi in Other Organizations
1.1.3 Results of Other Estimation Methods
1.1.3.1. GUMP Estimation Methodology
1.1.3.2. Implementation of GUMP Estimates
1.1.3.3. GUMP Historical Data
1.1.3.4. Accuracy of GUMP Estimates
1.2 Advantages of the Delphi Estimation Process
1.3 Disadvantages of the Delphi Estimation Process
2.0 DELPHI ESTIMATION PROCESS
2.1 ROLES AND RESPONSIBILITIES
2.1.1 The Delphi Estimation Team
2.1.2 Responsibilities
2.2 PROCEDURES
3.0 DOCUMENTATION
REVISION HISTORY
APPENDIX A
BIBLIOGRAPHY
GLOSSARY

LIST OF FIGURES

Figure 1. Delphi Estimation Form

Figure 2. Sample First Round Estimation Form

Figure 3. Sample Second (or subsequent) Round Estimation Form

LIST OF TABLES

Table 1. GUMP Estimation Accuracy

1.0 INTRODUCTION

1.1 BACKGROUND INFORMATION

1.1.1 History of the Delphi Estimation Process

The Rand Corporation developed the Wideband Delphi Technique in 1948. Its goal was to provide a process for achieving expert consensus without group conflict. Its initial purpose was the estimating of probable effects of a massive atomic bombing attack on the United States. It has subsequently been applied in the technological forecasting area and its use has proliferated in the United States and abroad.

The Wideband Delphi Technique has been applied to normative forecasting, ascertainment of values and preferences, quality of life issues, simulated and real decision making. Delphi has been applied to a process called "inventive planning" by which is meant the identification including invention of potential measures. These measures might be taken to deal with a given problem situation and the assessment of such proposed measures with regard to their feasibility, desirability, and effectiveness.

This procedure states that a group of experts is formed and asked not to discuss their work with one another. They give estimates of the product individually and anonymously. These estimates are collected and a coordinator assesses them. This assessment has to fall within an acceptable range, otherwise the coordinator asks for revised estimates, this process continues until a consensus is reached.

Overall, this process usually works well, but may be expensive and time consuming.

1.1.2 Employment of Delphi in Other Organizations

The Wideband Delphi technique is widely taught in universities and software organizations. After researching estimation techniques used by software organizations, we found a very interesting tenet. Several software organizations utilize the technique, without claiming the inheritance. Thus, even though there are few organizations who claim utilization of the Delphi technique, the methodology is widely prevalent. The Wideband Delphi Technique is also used by many organizations outside the software industry. Once again, it seems that the software industry is behind the power curve in adopting useful methodologies resident in other engineering organizations. Thus, our initiative to make the Wideband Delphi technique an integral part of GUMP.

1.1.3 Results of Other Estimation Methods

1.1.3.1. GUMP Estimation Methodology

GUMP estimates are presented in the form of man-hours for each tisk. The estimated man-hours are compared to the actual man-hours documented for tisk completion. Estimates are arrived at based on prior experience and best guess.

1.1.3.2. Implementation of GUMP Estimates

The GUMP software estimates are accomplished in process block 300 (Analyze) and are documented in the tisk analysis report. The estimates in the tisk analysis report typically cover detailed design, implementation, unit testing, and independent verification and validation. The actual man-hours are based on time card data and are recorded in the metrics summaries. The metrics summaries list man-hours for process blocks 500 (Implementation), 600 (IV&V), and 700 (Wrap Up).

1.1.3.3. GUMP Historical Data

The historical accuracy of GUMP estimates is shown in Table 1. The estimated hours were obtained from tisk analysis reports. The actual hours came from metrics summaries. The variance was calculated using the formula:

Variance = ((Estimate / Actual) - 1) x 100%

A variance of zero indicates the estimate exactly matches the actual. A positive variance indicates an overestimate. A negative variance indicates an underestimate. As the magnitude of the variance increases, the accuracy of estimate decreases. A positive variance represents the percentage of actual hours that were included in the estimate, but were not used. A negative variance represents the percentage of actual hours that were not accounted for in the estimate. In tisk T005, for example, 29 hours were estimated and 71.5 hours were actually expended. Therefore, 59.4 percent of the actual hours were not included in the estimate.


TISK ESTIMATE
( HOURS ) 		ACTUAL
( HOURS ) 		VARIANCE
(%)
T003 185.00 111.8 +65.5
T005 29.00 71.5 -59.4
T006 82.00 173.5 -52.7
T007 20.00 22.5 -11.1
T008 48.00 229.0 -79.0
T009 65.00 202.0 -67.8
T010 56.00 112.0 -50.0
T011 23.25 125.7 -81.5
T015 37.00 30.8 +20.3
T016 120.00 74.5 +61.1
T017 366.00 526.5 -30.5
T018 28.00 87.0 -67.8
T021 27.00 36.8 -26.5
T022 7.50 30.5 -75.4
T023 2.00 44.0 -95.5
T024 15.50 91.0 -83.0
T025 55.00 71.0 -22.5


Table 1. GUMP Estimation Accuracy.

1.1.3.4. Accuracy of GUMP Estimates

Based on the data in Table 1, GUMP estimates were off by an average of 55.9 percent. The estimates also tended to underrate the actual work required. In 82 percent of the cases the effort was underestimated. The historically high level of inaccuracy in the GUMP estimates indicates that adoption of the Delphi estimation method may be beneficial.

1.2. Advantages of the Delphi Estimation Process

1.3. Disadvantages of the Delphi Estimation Process

2.0 DELPHI ESTIMATION PROCESS

2.1 ROLES AND RESPONSIBILITIES

2.1.1 The Delphi Estimation Team

The Delphi Estimation Team shall consist of at least four estimators. The estimators will be, as a minimum:
  1. The Project Coordinator (PC)
  2. At least one representative from IV&V
  3. Optimally four Software Engineers, but as a minimum two Software Engineers
EXCEPTION: When Delphi is being used to estimate an IV&V process, the estimation team shall include the PC and all members of the IV&V team.

2.1.2 Responsibilities

PC: The PC shall be the single point of contact for the Delphi estimation documents dissemination and collection. The PC shall conduct the balloting process for each round of estimation. The PC will then forward the ballots to the Metrics section for data collection, maintenance, and analysis. The PC will also act as the tiebreaker, if the need arises.

IV&V: IV&V uses the estimation data to monitor the progress of the software engineering groups(s) in preparation for testing new changes. IV&V also uses the data to monitor their own progress and effectiveness at estimating their own testing requirements.

SQA: The SQA will use these metrics to ensure that the software engineers are maintaining a proper course and heading. If the engineers begin to vastly deviate from the estimation metrics, the SQA members will promptly bring this matter to everyone’s attention. This will permit the initiation of corrective action, or in extreme cases, a call for another Delphi Estimation.

SEPG: The SEPG will use these metrics to monitor the software engineering activities of the engineering groups. The SEPG will also maintain these metrics in the SEPG database. The SEPG will control and assist any changes necessitated by multiple rounds of the estimation process.

Software Engineers: The Software Engineers are responsible for using the resultant metrics as a guideline for timeline production, and work scheduling. They will also use these metrics to track their performance, and to inform the PC of any forthcoming problems.

2.2 PROCEDURES

Many variations of the Delphi technique can be created. The following steps outline our approach for using the Delphi technique:

  1. After sufficient familiarization with the tisk requirements, each estimator is given an estimation form as shown in Figure 1 of Appendix A.
  2. The estimators meet to discuss the tisk and any estimation issues.
  3. Each independently completes the estimation form.
  4. The estimates are given to the Project Coordinator, who tabulates the results and returns them to the estimators on an estimation form with the previous round's data filled in on the table as shown in Figure 3 of Appendix A.
  5. Only the estimator’s personal estimates are identified, all others are anonymous.
  6. The estimators may meet to discuss the results and revise their estimates as they feel appropriate.
  7. This cycle is repeated until the estimates converge to an acceptable range or until the completion of three cycles, whichever occurs first.

3.0 DOCUMENTATION

  1. The Delphi estimation process requires the use of an estimation form (shown in Figures 1 through 3, Appendix A).
  2. The tisk number, the estimator's name, and the estimation date should be entered in the appropriate blocks on the form. Each requirement is identified by number from the Analysis Report.
  3. Estimators provide estimates for each requirement on a separate estimation form.
  4. Estimates are based on the lines of code (LOC) and lines of documentation (LOD) to be inserted and deleted (Ins+Del) to implement the various requirements. Estimates for each requirement are entered in the LOC and LOD columns on the form(s).
  5. Person-hour estimates are made for each of the process blocks/cells in which development and testing work is to be done (cells 502, 503, 504, 505, and 601). Estimates may be given in whole hours or hours and tenths of hours.
  6. For each subsequent estimation cycle, the same estimation form is used. Statistics from the previous estimation cycle are provided for each requirement. These include:
    1. the estimator's previous estimate,
    2. other highest and lowest estimates of the other estimators, and
    3. the average estimate.
Additional space is provided for re-estimates of any or all of the estimator's previous estimates. If any re-estimates are made, the estimator may explain the rationale for each re-estimate at the bottom of the form.

REVISION HISTORY

February 2001
Bonnie Currey
Made changes to reflect new process cell numbers

April 1999
Joel Cook
Made Changes to Reduce Process Overhead

August 1998
Robert Penley
Converted Document to HTML Format

June 1998
Joel Cook, Shelly Wherley
Preliminary Document

APPENDIX A

Delphi Estimation Form

Name: _____________________________________          Date: _____________________

Tisk #: ___________________         Requirement #: _______________________
Note: Use a separate ballot for each tisk and/or requirement being estimated.

Estimation Round (circle selection):            1             2            3

 

LOC
Ins+Del

LOD
Ins+Del

502
(Hours)

503
(Hours)

504
(Hours)

505
(Hours)

601
(Hours)

Highest
Estimate

 

 

 

 

 

 

 

Average
Estimate

 

 

 

 

 

 

 

Lowest
Estimate

 

 

 

 

 

 

 

Your
Previous
Estimate

 

 

 

 

 

 

 

Your
New
Estimate

 

 

 

 

 

 

 

Note: First round estimates will have no data in the table above.
Estimators complete only the bottom row of the table in any round.

Estimator's Comments:




Figure 1: Delphi Estimation Form



Delphi Estimation Form

Name: __Joe Programmer_____________          Date: _____________________

Tisk #: __T038______         Requirement #: _2.1________________
Note: Use a separate ballot for each tisk and/or requirement being estimated.

Estimation Round (circle selection):            (1)             2            3

 

LOC
Ins+Del

LOD
Ins+Del

502
(Hours)

503
(Hours)

504
(Hours)

505
(Hours)

601
(Hours)

Highest
Estimate

 

 

 

 

 

 

 

Average
Estimate

 

 

 

 

 

 

 

Lowest
Estimate

 

 

 

 

 

 

 

Your
Previous
Estimate

 

 

 

 

 

 

 

Your
New
Estimate

85

15

65

10

2

30

5

Note: First round estimates will have no data in the table above.
Estimators complete only the bottom row of the table in any round.

Estimator's Comments:




Figure 2: Sample First Round Estimation Form



Delphi Estimation Form

Name: __Joe Programmer_____________          Date: _____________________

Tisk #: __T038______         Requirement #: _2.1________________
Note: Use a separate ballot for each tisk and/or requirement being estimated.

Estimation Round (circle selection):            1             (2)            3

 

LOC
Ins+Del

LOD
Ins+Del

502
(Hours)

503
(Hours)

504
(Hours)

505
(Hours)

601
(Hours)

Highest
Estimate

100

20

95

25

10

50

10

Average
Estimate

80

15

72.5

20

5

32

8

Lowest
Estimate

60

10

50

10

2

20

2

Your
Previous
Estimate

85

15

65

10

2

30

5

Your
New
Estimate

 

 

 

 

 

 

 

Note: First round estimates will have no data in the table above.
Estimators complete only the bottom row of the table in any round.

Estimator's Comments:




Figure 3: Sample Second (or subsequent) Round Estimation Form

BIBLIOGRAPHY

Boehm, Barry W., Software Engineering Economics, Prentice-Hall, New Jersey, 1981

Helmer-Hirschberg, O., Social Technology, Basic Books, New York, 1966

Hiltz, Starr Roxanne, Group-Based Estimation Methods: DELPHI

Humphrey, Watts S., Managing the Software Process, SEI Series in Software Engineering, Addison-Wesley, 1989

Schach, Stephen R., Classical and Object-Oriented Programming, Richard D. Irwin, Chicago, 1996

Tomayko, Dr. James E., Sizing Software http://www.cs.cmu.edu/~SW_Managemnt/html/mod_1/mod_1_3.html

Turoff, Murray; Hiltz, Starr Roxanne, Computer Based Delphi Processes http://eies.njit.edu/~turoff/Papers/delphi3.html


GLOSSARY

Ins+Del
Inserted and Deleted

IV&V
Independent Verification and Validation

LOC
Lines of Code

LOD
Lines of Documentation

PC
Project Coordinator

SEPG
Software Engineering Process Group

SQA
Software Quality Assurance