CHAPTER 3 Understanding the Decision Analysis Process

The Americans will always do the right thing … after they’ve exhausted all alternatives.

—WINSTON CHURCHILL

We want to make rational choices, we would like a transparent decision-making process, and we want a mechanism for correcting mistakes. These goals can be accomplished through the decision analysis process: decision framing, modeling, quantitative analysis and implementation, and monitoring and reviewing of decisions. Decision analysis is simple and adaptable to different types of project decisions.

DECISION ANALYSIS

An example of applying decision analysis to a complex project can be seen at the National Aeronautics and Space Administration (NASA). The U.S. Congress mandated that NASA be more accountable when it evaluates its advanced technology projects. As a public agency, NASA is concerned about maximizing the value it gets from its expenditures and cost reduction. But taking risks is inherent in all space exploration, so eliminating risk would mean eliminating NASA’s very purpose. (From another perspective, what do you think was the primary objective of the Wright brothers—maintaining a healthy net present value or getting an airplane to fly?) To manage its inherent risk, many NASA divisions have started applying decision analysis methods to improve their ability to select courses of action.

Once a year, divisions of the Kennedy Space Center and their contractors submit as many as 50 project proposals for possible funding. To choose among them, the Space Center uses a multi-criteria decision-making process for evaluating and prioritizing advanced technology projects (Tavana 2003). Project evaluations are done by committees with 15 members: five permanent voting and ten advisory (it sounds like the U.N. Security Council).

In the first phase of the process, the five permanent members of the committee, who are appointed by Kennedy Space Center management, perform a preliminary review of the projects. They identify which stakeholder departments—such as safety, system engineering, cost saving, process enhancement, and reliability—should participate in the review. Ten representatives of the stakeholder departments then join the committee as advisory members. The committee decides what criteria should be considered in evaluating the project. Examples of these criteria are “Eliminating the possibility of death or serious injury,” “Reducing a system failure,” “Improving time to repair,” and “Meeting proposed schedule.”

The committee assigns certain weighted coefficients to each criterion. It then conducts brainstorming meetings to determine the probability of occurrence for each criterion in each project. Quantitative analysis tools are then employed to analyze the projects and rank them. Finally, the committee submits recommendations for the approval of top management.

The decision analysis process at the Kennedy Space Center does not replace human judgment in project selection. But, importantly, it enables decision-makers to think systematically and, as a result, improve the quality of their decisions.

WHEN DECISION-MAKERS GO BAD

Everybody wants to make good decisions, but in reality many project decisions turn out to be wrong. For example:

Why did a company decide to develop a product that it could not sell?

Why did a project team decide to use supplies of poor quality?

Why did management appoint a project manager who does not understand the business?

You might say that the company did not really want a product it could not sell, did not really wish to use poor-quality materials, or did not really intend to appoint an incompetent project manager. But it did make those decisions, using its own decision-making process.

The process that leads to such decisions is often cloaked. Decisions are frequently made behind closed doors, so it is difficult to know how, when, where, and why a decision that had dire consequences was made. And once people make a decision, they tend to stick to it, even in the face of mounting evidence that it is wrong. This is a known psychological bias. People tend to be consistent, even if it means behaving irrationally; they don’t what to admit that they made a bad decision. As a result, poor decisions tend to take on a life of their own, which can exacerbate an already bad situation.

But there is a better way—decision analysis.

THE DECISION ANALYSIS MANIFESTO

Here are three basic things you should want from a decision analysis process. We call it the Decision Analysis Manifesto.

You want rational decisions. Quality decisions should lead to maximizing project value while minimizing expenditures. Good decisions should be based on an unbiased assessment of all possible alternatives. They should benefit the whole business, not just the interests of a specific individual or group.

You want transparent decision-making. You want to know how decisions are made, who made them, and who should be accountable if a decision is wrong. You want to be able to participate in decision-making.

You want a mechanism for correcting mistakes. If there is a problem with the original decision, you should be able to recognize it and take corrective actions in a timely manner.

In sum, what you want is a process. As with many other business processes, decision analysis has a set of procedures and tools that an organization can readily follow.

THE “3C” PRINCIPLE OF PROJECT MANAGEMENT

Decision analysis is not a single, fixed process, but rather an adaptable framework that can be tailored by an organization to meet its specific needs.

There is no exact recipe for how to structure a decision analysis process. Different processes can be adopted for various companies, types of projects, and the types of decisions required. Your organization may already have, if not a fully established process, some components of decision analysis. For example, how do you review project risks and uncertainties? How are decisions made at product launch meetings?

The decision analysis process is an integrated set of procedures, rules, preferences, and tools that help the organization make a rational choice.

But what does a “fully established decision analysis process” imply? Any established or mature decision analysis process is based on three main rules, which we call the “3C” principle: consistency, comprehensiveness, and continuity.

Consistency

The decision analysis process should be standardized for similar kinds of problems and opportunities. Inconsistency in decision-making can cause projects to change directions unnecessarily, which can lead to failure. This necessitates that organizations must have the same set of rules and preferences for making decisions in all similar types of projects.

Suppose, for example, an oil company has different offices around the world evaluating exploration prospects. The company does not have enough resources to drill everywhere at the same time, so it must make choices. The evaluations on potential drilling prospects are forwarded from the various outlying offices to the corporate planning headquarters, where decisions about resource allocation are made. One of the main difficulties that the corporate planners face is that this information is submitted by different groups looking to develop their prospects in different locations, which are often in different countries. The planners don’t want to compare apples and oranges, as the saying goes, so the company tries to ensure that the methods used to generate the data regarding prospects are consistent across the organization. Otherwise it would be impossible to make a comparison of potential oil reserves and make decisions on which prospects to develop (Rose 2001).

Comprehensiveness

Decision analysis processes should include a comprehensive assessment and analysis of the business situation. Missing or incomplete information can lead to incorrect decisions.

Let’s say that your manager approaches you and shows you a project schedule. He says, “We performed a comprehensive analysis on all of our possible alternatives and have decided to go ahead with this particular one.”

After a brief look at the schedule, you say, “Looks like it’s a very tight schedule. Did you account for any risk events? Where is the contingency time?”

“Don’t worry about contingencies,” he replies. “We covered everything. If you recall, we have done some similar projects in the past and didn’t have any problems. Plus, the decision has come down from up high, so it’s out of our hands.”

Two days into the project, a major event occurs and the project is delayed.

Did upper management perform a comprehensive decision analysis as claimed? No, because they relied simply on their experience of some past projects and did not include risks and uncertainty. The analysis was not comprehensive, and therefore it was flawed.

Continuity

Decision analysis is a continuous process of evaluating and refining decisions during the course of a project. High-quality results can be achieved only through constant and consistent adaptive management.

Here’s an example of failing to use adaptive management.

You recently completed a project and presented it to your client. Unfortunately, the client is not very happy. Why? A couple of years ago, when the project was initiated, somebody made an incorrect decision that appears to have reduced the value of the project due to huge cost overruns. The person responsible for that decision is no longer on the team, so it will be difficult to understand the basis for it. And why were no corrective actions taken when it became obvious that the evaluation of cost was flawed?

Upon further investigation, you discover that it is not a common practice in your organization to revisit a decision once it has been made. In addition, when the problem surfaced, a great deal of resources had already been expended, and there was a great reluctance at that point to change course. The final result is that your project failed to meet its requirements because the team did not adapt to changing circumstances.

DECISION ANALYSIS PROCESS VS. THE PMBOK® GUIDES PROJECT RISK MANAGEMENT

You may question how decision analysis differs from what is described in the PMBOK® Guide, Chapter 11, “Project Risk Management.” Decision analysis and risk management both deal with uncertainties, but they do so from different perspectives. So are both processes required? Could you imagine a country with two presidents—one responsible for decision-making and the other for managing risks? Just think, two White Houses, two sets of Secret Service agents, and two White House chefs. This might not be the end of the world, for we have a lot of redundant government agencies and we could survive with two White Houses. However, the two presidents would have a difficult time with their appointed roles: You can’t make a decision without risk analysis, and you can’t manage risks without making decisions.

In many aspects, decision analysis enriches the project management process described in the PMBOK® Guide.

The good news is that you do not need two (or more) business processes to manage related issues. You can integrate decision analysis into your overall project management process as described in the PMBOK® Guide. Decision analysis and risk management processes have a significant overlap. A good decision analysis should include risk analysis. In other words, decision analysis is a broader exercise than risk analysis.

In this book we do not discuss the PMBOK® Guide processes in detail. (A number of very good books on risk management are listed in the Future Reading section.) We will, however, show you how the PMBOK® Guide‘s risk-management processes parallel the decision analysis. We will mention all of the PMBOK® Guide‘s risk-management processes, so you will have a complete, logical picture of how the two approaches relate to each other.

DECISION ANALYSIS AND OTHER BUSINESS PROCESSES

The PMBOK® Guide‘s project management process is not the only process practiced in organizations. Many organizations have established quality management procedures and use different methodologies to improve quality, performance, reliability, and customer satisfaction.

One of these methodologies is Six Sigma (Pande and Holpp 2001; Pyzdek 2003), a rigorous and disciplined methodology that uses data and statistical analysis to measure and improve a company’s operational performance, practices, and systems. Six Sigma identifies and prevents defects in manufacturing and service-related processes.

Many organizations have spent significant efforts implementing ISO 9000 (Hoyle 2005; Gupta 2006). ISO 9000 is a family of the ISO (the International Organization for Standardization) standards for management systems. It does not guarantee the quality of end products and services; rather, it certifies that consistent business processes that fall within the ISO 9000 guidelines are being applied.

Software development organizations use different business processes, including the Rational Unified Process (RUP) (Kruchten 2000; Larman 2002). The Rational Unified Process is an iterative software development process created by the Rational Software Corporation, now a division of IBM.

Because decision analysis is an adaptable framework, it can be integrated into these different business processes; for example, it can share the same data and analytical methods.

PHASES OF THE DECISION ANALYSIS PROCESS

The decision analysis process includes a number of phases, with each phase containing a number of steps. We can illustrate the process with a project you may be familiar with from your childhood, an incident involving Winnie the Pooh:

As you may recall, Pooh dropped in on Rabbit one day and ended up jammed in Rabbit’s doorway after helping himself to all of Rabbit’s honey. For Pooh, it was supposed to be a very short project: (1) visit Rabbit; (2) consume honey; and (3) go home. But Pooh, being Pooh, ate too much honey during the “consume honey” activity. This is a good example of the psychological bias of overconfidence. As a result of this event, the trivial activity (“go home”) could not be accomplished as scheduled, for Pooh was firmly wedged in the doorway. Now Pooh and his friends had a decision to make: They had to select the best alternative to solve this problem (Figure 3.1).

Let’s examine how a decision analysis process (Figure 3.2) would help identify the best solution for the problem facing Rabbit and Pooh.

Phase 1. Decision-Framing

Decision-framing helps decision-makers identify potential problems or opportunities; assess business situations; determine project objectives, tradeoffs, and success criteria; and finally identify uncertainties. The project manager defines the scope of the decision. Depending on the situation, a project manager alone, an independent expert, or a team of experts can perform the decision framing. The processes of risk-management planning and risk identification described in the PMBOK® Guide can be accomplished along with decision-framing.

Step 1.1. Identifying Potential Problems and Opportunities

In some cases it is difficult to identify problems and opportunities, especially when they are related to a strategic decision. For example, what causes different projects within the organization to be consistently late?

Figure 3.1 Using Decision Analysis to Resolve Complex Problems

In our Pooh example, the problem was clear. Pooh was stuck and was not happy about it (neither was Rabbit). Both of them need Pooh to be removed from Rabbit’s house as soon as possible.

Step 1.2. Assessing the Business Situation

Before making a decision, it is important to assess the business environment and define the constraints related to the problem. Business environments can influence resource availability and costs. The assessment may also include an analysis of markets, competition, prices, or anything else that can be related to the problem or opportunity. During this step, it is important to list all external factors that may have an impact on the problem.

Who or what could be used to get Pooh out of his predicament? Of course, it could be Christopher Robin and Pooh’s other friends. Wise Owl also had some project management experience. In addition, Gopher had the expertise and tools to provide some engineering work.

Figure 3.2 The Decision Analysis Process

Step 1.3. Determining Project Objectives, Tradeoffs, and Success Criteria

Projects usually have multiple objectives and therefore multiple criteria for decision-making, which can make the analysis very complex. Decision-making criteria include project duration, cost, scope, quality, and safety, among other parameters. Project managers should find the right balance between these objectives and make tradeoffs when necessary.

In Pooh’s situation the success criteria were:

Remove Pooh from the doorway as soon as possible.

Do not harm Pooh during this process (safety concern).

Do not damage Rabbit’s dwelling.

Step 1.4. Identifying Uncertainties

Understanding uncertainties is the key to the decision analysis process. In the decision-framing step, risks and uncertainties should be identified. Uncertainties can be found in a project’s cost, scope, duration, quality, safety, or environment.

In this project—removing Pooh—we primarily have uncertainties in time, as well as uncertainties in cost.

Step 1.5. Generating Alternatives

In decision-framing, it is important to generate key alternatives. First, you must identify what cannot be changed, that is, what the constraints are in the particular decision analysis. Then you can determine potential alternatives.

Pooh needed to be removed one way or another. He could not be stuck in Rabbit’s doorway forever. Therefore, project scope was a constraint. There was, however, the possibility of bringing in additional resources to accelerate the project. As a result, we have three potential project scenarios:

External contractor Gopher digs out Pooh.

Gopher blasts Pooh out with dynamite.

Christopher Robin’s suggestion—waiting until Pooh loses enough weight and is slim enough to slip through the doorway.

Phase 2. Modeling the Situation

A mathematical model can help to analyze and estimate future events. To understand how a structure withstands particular loads, engineers perform structural analysis on buildings using mathematical models. They don’t want to find out that a particular beam cannot bear a load after it has been put in place because by then it may be too late to change it. The same situation exists with any other projects.

Step 2.1. Creating Models for Each Project Alternative

Project managers constantly create mathematical or valuation models of projects. In most cases, it is simply the project schedule. A more comprehensive model of a project includes a breakdown of resources, costs, and other project variables.

Sometimes, quite elaborate models are required. For example, in the analysis of a product’s lifecycle, comprehensive models will include not only product development but also marketing and sales efforts.

Each model includes input and output variables, as well as a calculation algorithm. In the case of a project schedule, input parameters are tasks with their start and finish times, costs, and resources, and relationships among all the tasks. Outputs include project cost, duration, start and finish times, critical tasks, resource allocation, and other parameters.

Modeling the situation in general, and project scheduling in particular, can be a very complex process. A model should be created for each project scenario; in most situations, however, these models are variations of a single base model.

In the Pooh removal project, a schedule for each alternative was needed.

Based on Owl’s request, Gopher estimated the duration of the excavation alternative. He did a review of the site and performed some exploratory excavation. He estimated that the work would take two or three days. He also performed a cost analysis. He based his calculation on his hourly rate and estimated project duration. He also added overtime and 10% for contingency.

Gopher estimated that using dynamite would lead to a quick removal of Pooh, but with uncertain effects on Rabbit’s doorway and Pooh’s rear end.

The slimming alternative, suggested by Christopher Robin, seemed to have the least risk and cost, but the longest duration.

Step 2.2. Quantifying the Uncertainties

The uncertainties in a project, identified through the decision-framing process, should be quantified. One of the ways to quantify uncertainties is to define ranges for their parameters. For example, define low (optimistic), base (expected), and high (pessimistic) duration or cost estimates for each task.

Another way to define uncertainties is to list all the potential events that could affect the project schedule, then quantify their probabilities and impact.

The PMBOK® Guide‘s steps for risk management recommend using a qualitative risk-analysis process to assign probability and impact.

Gopher estimated the uncertainty in duration of the excavation as a range (between two and three days).

The blasting alternative had minimal uncertainties in estimating duration.

There were several uncertainties with the last alternative (Pooh’s slimming down). Nobody knew for certain how long before Pooh’s stout frame would melt away enough to free him from the door. They were also faced with the prospect that Pooh would continue to eat (on the sly) during the course of the project. That risk, which should be given both a high impact and a high probability, could significantly increase project duration.

Phase 3. Quantitative Analysis

After the mathematical model is ready, the analysis may include a number of steps, depending on the situation. It is possible to apply simulation techniques to analyze the project. These techniques can give project managers enough data to make an informed decision.

Even with the most advanced analytical tools and techniques, interpretation of the results of an analysis is performed subjectively by project managers; therefore, it is subject to the types of mental traps discussed in Chapter 2. For example, quantified analysis may show that the chance that a project will be completed on time equals 80%. Is this acceptable? It may be in some projects, not in others. In addition, the results of quantitative analysis must be communicated properly to decision-makers to minimize the potential for biased decisions.

Quantitative methods such as sensitivity analysis, Monte Carlo analysis, and decision tree analysis are described in Chapter 11 of the PMBOK® Guide.

Step 3.1. Determining What Is Most Important

A model of a project can include a considerable number of variables: large numbers of tasks, resources, risks, and other parameters. Some of these parameters may significantly affect the course of the project. For example, certain risks will cause failure of the project, whereas other risks will have no noteworthy effect. It is impossible for a project manager and a project team to concentrate their efforts on mitigating all possible risks. The team should therefore focus first on mitigating critical risks.

To determine which project parameters are the most critical, project managers can use sensitivity analysis, which is described in detail in Part 4 of this book.

In the Pooh situation, the risk associated with feeding Pooh would probably have the most effect on the project duration and was therefore a critical risk. To mitigate this risk, Rabbit set up a poster, “Do Not Feed Bear.”

Step 3.2. Quantifying Risks Associated with the Project

Uncertainties associated with input parameters were already quantified during the modeling step. Now you need to analyze how the combination of all these uncertainties could affect the project. The goal of this analysis is to create a “risk profile” of the project. You may need to know the following information:

The chance that the project will be completed within a certain period and on budget

The project’s success rate, or the chance that it will be completed

The low, average, and high estimates for duration, cost, and other project parameters.

Again, a number of analytical techniques can be applied to this analysis.

Here is the result of the analysis of Pooh’s project based on the success criteria identified during the decision-framing stage. There were three alternatives:

“Excavate Pooh.” There was a 100% chance of damaging Rabbit’s home, a significant chance of harming Pooh, and a very significant chance that the project would be completed within a few days.

“Blast out Pooh.” There was a 100% chance of damaging Rabbit’s home, a very significant chance of harming Pooh, and a very significant chance that the project would completed almost instantly.

“Slim down Pooh.” There was zero chance of damaging Rabbit’s home, zero chance of harming Pooh (although he might have an extended period of relative deprivation), and large uncertainties in the project duration.

Step 3.3. Determining the Value of New Information

A useful decision analysis technique is to assess the value of new information.

For example, Gopher needed to estimate the duration of the excavation project to remove Pooh from the doorway. Gopher could perform exploratory excavation, but it could be costly and time-consuming. This analytical technique helps establish the value of new information—how much money can be saved if additional information is obtained through an exploratory excavation.

Step 3.4. Deciding on a Course of Action

In some cases, it is easy to select the most efficient alternatives based on the results of risk analysis. The best alternative based on selected success criteria may be obvious and can be easily selected without further steps.

In many situations, however, the selection between the alternatives is not so easy. Sometimes, different scenarios are executed only if certain conditions exist. In many cases, decisions are made using numerous criteria, which complicates the selection of the most efficient alternative.

In the Pooh example, the decision was based on multiple criteria. The safety of Pooh was the first priority; therefore, the blasting alternative had to be rejected. The excavation alternative was also rejected because it did not provide adequate safety and could cause substantial damage to Rabbit’s house. Therefore, despite concerns about project duration, the slimming-down alternative was selected. Interestingly, in a Russian cartoon version of the same Pooh story made in the early 1970s, Pooh’s friends and Pooh himself decided that he would not be able to withstand the deprivation imposed by the slimming alternative, so they just yanked him out, causing a great deal of structural damage to Rabbit’s house in the process. Does this say something interesting about the differences between Western and Soviet psychology? Or perhaps the producers of the cartoon were on a limited budget and decided that having the characters forcefully dislodge Pooh was cheaper to produce and therefore a better choice.

Phase 4. Implementation, Monitoring, and Review

Phase 4 involves two steps.

Step 4.1. Project Implementation and Monitoring

Now the decision has been made, and the selected course of action is under way. But what if, in the middle of the project, an unforeseen event occurs, causing the plan to deviate? Luckily, there is a valuation model of the selected project alternative (created in step 1.5), and the project manager can update the model, perform a new analysis, and make a decision. It is very important to track project performance constantly and analyze all potential pitfalls and opportunities. In many cases, a new decision-framing step will be required if a new decision within the same project is required.

Tracking project performance helps you to forecast what could happen to the project, even if some activities are only partially completed. Before the project started, you had only one source of input information for the decision analysis process: historical data, which is either objectively defined based on certain records or is the result of expert judgment based on past experience. Now the project manager can also use actual project performance to make decisions. This process for continually improving decisions by learning from the outcomes of earlier decisions is why adaptive management is so powerful.

Pooh’s friends continually checked Pooh’s slowly shrinking girth, trying to estimate when he would be slim enough to pop out. Eventually, when their measurements indicated that the time was ripe, they managed to extract Pooh without damage to either Rabbit’s home or Pooh himself.

Step 4.2. Review of the Decision Experience

You need to know whether your analysis and decisions were correct. Otherwise, you will make the same mistakes all over again.

Apparently, in this situation, the decision was correct. Some small things could have been done better. For example, the sign “Do Not Feed Bear” could have been installed at the beginning of the project rather than after Gopher’s offer of food to Pooh.

BIG AND SMALL DECISIONS

When we ask project managers why a decision analysis process has not been implemented in their organizations, they usually give some form of the following replies:

We don’t know what the decision analysis process means.

We already have processes, such as project management, and we don’t need to introduce yet one more.

Decision analysis processes are not suitable for our organization or our types of projects.

Our projects are small, and we make decisions based on our experience and intuition.

We are planning to implement the process in the future, but we are too busy right now.

Decision analysis processes require significant resources, including training that we don’t want to invest in right now.

Interestingly, most project managers use a number of answers from this list instead of just one. However, the third answer (“Decision analysis processes are not suitable for our organization or our types of projects”) happens to be most common. Most managers believe that decision analysis is suitable only for large projects. In their opinion, intuitive decision-making is sufficient for small projects. But what differentiates a small project from a large one? Is drilling one well that costs $2 million a small project for a large oil company? Maybe. But a $2 million software project is definitely a large project. So it is all relative to the particular industry or organization.

Rule number one in project decision analysis: The process must be simple.

We agree that small projects probably do not require as complex an analysis as large projects. But we strongly advise against relying purely on intuitive decision-making for a project of any size.

Table 3.1 offers some advice on how you can tailor the decision analysis process to different types of decisions:

In any case, remember that the process should be as simple as possible. If you suggest a full-blown process for your organization, and your organization is not ready for it, it will kill the whole idea (and any hope we have of selling more books to your colleagues). In addition, remember that in 99 percent of cases, project decision analysis is simply an exercise in rational thinking.

Table 3.1 Decision Analysis Process for Different Types of Projects

THE VALUE OF PROJECT DECISION ANALYSIS

We often hear this question: “Does a decision analysis process bring any benefits or it is just extra paperwork?”

Clemen and Kwit (2001) studied the use of decision analysis at the Eastman Kodak company. They analyzed the decision-making in 178 projects that were carried out over a period of ten years. The projects included new product brainstorming, vendor selection, emission-reduction analysis, process analysis, and others. The project durations varied from 20 hours to one year. Clemen and Kwit were trying to determine an incremental dollar value generated by decision analysis. In most cases it was hard to measure the actual value added to Eastman Kodak’s business, but their estimate was that for all the projects combined, decision analysis added more than $1 billion in value. The average value per project was between $6.65 million and $16.35 million. In addition to the monetary benefits, the decision analysis added value by promoting careful thinking about strategies, facilitating discussion among stakeholders, and providing a common framework for risk analysis and decision-making within the company. Clemen and Kwit concluded that even though it was hard to measure value in specific projects, decision analysis brought substantial value to the company.