Predicting a defect inflow is important for project planning and monitoring purposes.

For project planning purposes and for quality management purposes, an important measure is the trend of defect inflow in the project i.e. how many defects are reported in a particular stage of the project. Predicting the defect inflow provides a mechanism of early notification whether the project is going to meet the set goals or not. In this paper we present and evaluate a method for predicting defect inflow for large software projects: a method for short-term predictions for up to three weeks in advance on a weekly basis. The contribution of this paper is the fact that our model is based on the data from project planning, status monitoring, and current trends of defect inflow and produces results applicable for large projects. The method is evaluated by comparing it to existing defect inflow prediction practices (e.g. expert estimations) at one of the large projects at Ericsson. The results show that the method provides more accurate predictions (in most cases) while decreasing the time required for constructing the predictions using current practices in the company.

Chapter- 8: PROJECT

EVALUATION
Significance Purpose of Evaluation Types of Evaluation Difference between Monitoring and Evaluation Evaluation Indicators Requirements for Undertaking an Evaluation Exercise Feed-back for the Future

Significance
8.1 Planning and Development Division, which occupies the central position in the planning machinery in Pakistan, also assumes the role of project evaluator. As per the Rules of Business, 1973 (updated upto 1985), the evaluation of on-going and completed projects is one of the basic responsibilities of the Planning and Development Division. The Planning and Development Division (Projects Wing) while submitting a summary to the NEC on "Improving the Efficiency of Development Expenditure through Institution of Appropriate Monitoring and Evaluation Procedures" on 30th June, 1988, brought out that post-completion evaluation is necessary for improving the effectiveness of development expenditure which had largely been ignored. The NEC while taking note of the summary has not only accorded approval for creating/strengthening the Project Monitoring and Evaluation Cells (PME) in the Federal Ministries/Divisions, introduction of computers for evaluation purposes and enhancement of the status of the Projects Wing to that of a Division but also directed the Projects Wing to enhance project

evaluation studies from 10 to 50 studies annually. A proforma designed by the Projects Wing for project evaluation is enclosed at Annexure-XLVI.
Go To Top

Purpose of Evaluation
8.2 The final phase in the project cycle is project evaluation. The analyst looks systematically at the elements of success and failure in the project experience to learn how to plan better for the future. The basic objective of such a study is to ascertain the real worth of a project or programme as far as possible. Broadly speaking, evaluation may be defined as "a process which attempts to determine as systematically and objectively as possible the relevance, effectiveness and impact of activities in the light of the objectives". It is, thus, a critical analysis of the factual achievements/results of a project, programme or policy vis-avis the intended objectives, underlying assumptions, strategy and resource commitment. In specific terms, it makes an attempt to assess objectively the following:(a) the relevance and validity of the objectives and design of the project/programme in terms of broader issues of development policy, sector/sub-sector priorities and strategies as well as other problems of a wider nature; (b) the efficiency and adequacy of the pace of progress of the project/programme where the focus is mainly on managerial performance and productivity; (c) the effectiveness of the project/programme - a major part of an evaluation exercise-in realizing the intended objectives from a variety of angles; and (d) the identification of reasons for the satisfactory or unsatisfactory accomplishment of the results of the project/programme and to deduce critical issues and lessons which may be of relevance to other on-going and future projects/programmes of a similar nature.
Go To Top

Types of Evaluation
8.3 Evaluation, can be applied for different purposes as well as to a specific activity, project or programme. It is not restricted to the completion stage only but involves periodic investigations at many stages. The different types of project evaluations carried out are: (i) ex-ante evaluation, (ii) on-going evaluation and (iii) terminal evaluation/ex-post evaluation. The ex-ante evaluation/pre-approval appraisal has already been discussed with methods and techniques in Chapter-5. The on-going evaluation is carried out by the organization of its own to re-assess the projected feasibility of the PC-I content because of the time lag, while external evaluation is done by an agency other than the body involved in the implementation of a project. Ongoing and post-completion evaluation are discussed below:(a) On-going/Mid-term Evaluation The main purpose of an on-going/mid-project evaluation is to assist the project management to make appropriate adjustments in the changed circumstances or to rectify any shortcomings in the original design, so as to improve its efficiency and overall performance. (b) Post-Completion Evaluation The purpose of an ex-post or post-hoc evaluation is to discover the actual, as opposed to the projected, results of implementing a project. The aim of evaluation is primarily to compare the actual outcome of the project with the projections made at the appraisal stage. The examination of different aspects of the project can provide important lessons derived from experience for the new projects. The overall impact of the project will result in a number of effects which can be classified as costs and benefits, direct and indirect or tangible and intangible. Ex-post evaluation takes place after the completion of the project and is often more in-depth as it focuses on the analysis of impact. Besides, it is time-consuming, costly and calls for persons with special skills.
Go To Top

Difference Between Monitoring and Evaluation

8.4 Evaluation is a learning management tool but differs materially from monitoring. Project monitoring is undertaken at the implementation stage while evaluation is generally preferred when a project is complete. The monitoring reports provide the data base for the "evaluation" but evaluation cannot contribute directly to monitoring. The evaluation studies are more comprehensive in nature, covering all aspects of the projects, whereas monitoring provides information mainly to assess and help maintain or accelerate the progress of implementation. However, key differences between M and E functions are summarized below:

Monitoring (M)
Keeps track of daily activities a continuous function. Accepts objectives, targets and norms stipulated in the project document Checks progress towards output targets Stresses conversion of inputs to outputs Reports on current progress at short intervals for immediate corrective actions

Evaluation(E)
Takes long range view through indepth study - a one time function Questions Pertinence and validity of project objectives / targets Measures performance in terms of objectives Emphasizes achievement of overall objectives Provides an indepth assessment of performance for future feedback
Go To Top

Evaluation Indicators
8.5 Evaluation indicators are the yardsticks for the assessment of overall performance of a project/programme with reference to stipulated targets and objectives. The main indicators can be identified as under:i) Physical achievements indicators Overall physical progress Overall cost utilisation Timely or untimely completion of a project or a programme (delay in years). ii) Output or Impact Indicators Production (whether crops, livestock, forest products, fish, etc.) e.g., percentage of children in a target group receiving supplies feed, number of acres surveyed, loan applications processed/approved, trained manpower, a laboratory set-up etc. iii) Economic Indicators Financial and economic benefits (e.g., financial rate of return, internal rate of return, benefit-cost ratio, etc.). iv) Social Indicators - Quality of Life Indicators Income distribution with equity, level of food consumption, health and education facilities, shelter, access to essential amenities/basic needs, life expectancy, etc.
Go To Top

Requirements for Undertaking an Evaluation Exercise

8.6 In order to undertake an evaluation of any project/programme, a study of its related documents is a pre-requisite for acquiring sufficient knowledge and information thereof. The documents include:-

(a) The approved PC-I alongwith the related project documents, concept clearance papers loan/grant agreement with foreign agencies etc, and feasibility study, if any;
(b) Pre-approval appraisal notes/CDWP working papers; (c) Pre-approval technical scrutiny notes; (d) ECNEC summary and its decisions; (e) Sources of financial and other inputs; (f) Annual/quarterly progress reports; (g) Project review/monitoring/mid-term evaluation reports; (h) Special reports; and (i) Project completion report. The aforementioned documents form the basis of the assessment of different project activities. The site visits to the project will also assist in the assessment of the quantitative aspects of a project and its effect/ impact on the target group/beneficiaries. The following facts should be looked into in any evaluation exercise: i) Was the project properly conceived? Has it fulfilled its basic objectives? If not, what was wrong with the basic design? ii) Was the project adequately prepared? Were the forecasts of output or benefits correctly made to a reasonable extent? Was the technical preparation adequate? How good were the original cost estimates? If there were deficiencies in preparation, how those were removed or could be made up? iii) Was the project implemented as per the plan? If not, was this because of its haphazard preparation or because of delays in (a) the authorisation procedure, (b) obtaining suitable funds, and (c) other reasons? What lessons could be learnt to improve the implementation of other projects?
Go To Top

Feedback for the Future

8.7 Feed-back is the most important element of a systematic and integrated approach towards project appraisal, monitoring and evaluation. Essentially, it is the evaluation exercise which provides lessons for the feed-back, because the main objective of such studies is to compare the actual outcome of the project with the projections made in its appraisal and then the examination of essential positive and negative effects of the project, providing important lessons for the future. The feed-back from evaluation is a basic requirement of the management. An evaluation without any direction or support from the management can hardly be meaningful. To promote feed-back from evaluation, it is necessary to (a) substantiate proper evaluation findings and pay proper attention to specific issues of substance, (b) establish a feed-back mechanism, preferably to the policy-makers and senior management, and (c) rely upon feed-back through formal and informal arrangements. Feedback from the evaluation is used for operational (mid-course corrections and follow-up action), analytical (improvement of project design, objectives etc.) and policy purposes (finding out the validity of a given development strategy etc.). To ensure that feed-back is used in systematic manner, there must be an adequate institutional mechanism for channelling the findings and recommendations to the appropriate decision-makers and managers for the necessary follow-up action. The managers and policy-makers should know how to guide and use evaluation for their needs.