Site management options: federal contaminated sites decision-making framework

• Previous page
• Table of contents
• Next page

Appendix B: site management options assessment

This appendix discusses methods for assessing the relative advantages and disadvantages of a variety of remediation or risk management (R/RM) options.

Part 1: the theory

Role of cost/benefit analysis

Cost/benefit analysis is a commonly accepted approach for determining the feasibility of various alternatives being considered to address a particular problem or project requirement. In the context of contaminated sites, this can be used as a basis to determine the optimum approach where a variety of alternatives exist to address site conditions, representing a range of effectiveness, implementability and cost considerations.

In order to conduct the cost/benefit analysis, it is typically necessary to determine the alternatives for consideration, identify the evaluation criteria to be applied, and then conduct the comparative evaluation using an appropriate method. This is discussed in more detail below.

Development of an alternatives array

The formulation of a range of alternatives can be illustrated in an array that identifies the technical options available for addressing the site conditions (identified prior to Step 7). This may include both remediation and risk management techniques, and a combination depending on the site-specific circumstances and the environmental media that need to be addressed.

For illustration purposes, an example alternatives array is included in Table 1. As shown in the table, a total of nine alternatives are included. Each alternative includes specific actions related to individual site-specific areas or units (e.g., equipment/storage vessels, waste disposal areas, impacted soil areas, impacted groundwater areas). The scope of the alternatives progress sequentially from “do nothing,” to limited action (waste material removal, access restrictions, monitoring), then containment, and finally to active remediation through treatment. The potential application of both risk management and remediation methods is an integral part of the array, and a variety of other combinations may be available beyond what is shown in this example. In carrying out the comparative assessment, it may ultimately be decided that the optimum alternative is a refinement of the alternatives initially included in the array. The evaluation process should provide flexibility for this type of decision making.

It is noted that Alternative 1 (no action) may appear inherently unacceptable at the outset. However, its inclusion in the array may be useful for representing a baseline condition to identify the consequences of “do nothing” and for assisting with the justification for selecting one of the other alternatives.

Table 1: example alternatives array

Potential evaluation criteria

In most situations, the comparative evaluation of alternatives relies on the use of criteria against which each alternative can be assessed relative to other alternatives. These criteria may be either qualitative or quantitative, and will generally consider factors relating to overall protection, effectiveness, implementability, cost, stakeholder considerations and regulatory compliance. A list of potential evaluation criteria is included in Table 2. As shown in the table, various criteria may be applicable under each category, and are further classified according to the type of criteria (threshold, balancing and acceptance).

Table 2: potential evaluation metrics

Threshold criteria include minimum requirements that need to be met in order for the alternative to be considered for selection. Typically, this relates to protection of human health and the environment, and regulatory compliance. Alternatives that satisfy these threshold criteria will be suitable for passing an initial screening and then short-listed for more detailed consideration.

Balancing (or modifying) criteria are those which are used to compare the relative merits of the various alternatives and associated trade-offs. For example, Alternative A might be expected to achieve a permanent solution and unrestricted future site use, whereas other alternatives may achieve an acceptable cleanup level for commercial or industrial site use at a much lower cost.

Acceptance criteria are those that relate to meeting the expectations of various stakeholder groups, including various levels of government and the local community. Acceptance considerations may ultimately be used to make a final selection from a list of several otherwise suitable alternatives, or as a basis for refinement of a preferred alternative.

An alternative that is preferable based on balancing criteria considerations may not ultimately be selected for implementation if it is not acceptable to stakeholders. In some cases (e.g., large/complicated sites), stakeholder input will probably have already been considered in developing the list of alternatives.

The list shown in Table 2 represents potential criteria for consideration. The selection of relevant criteria may be affected by site-specific factors, and it may be decided that some should be deleted, or others added, as necessary and appropriate.

Options for comparative evaluation of alternatives

A variety of methods are available for conducting a comparative evaluation of alternatives in order to identify the most suitable alternative (and hence site management strategy) for implementation. This includes the following examples described herein: ad hoc methods, checklist methods, economic methods, pairwise comparison methods and matrix methods.

Ad hoc methods compare alternatives in narrative terms without using any explicitly stated methods to order the preferences, based on professional judgment. Typically, the use of ad hoc methods, as the name implies, does not necessarily follow an explicit set of evaluation criteria. This approach can be applied to situations in which the scope of the problem is narrow and well defined, and the rationale for selection of the proposed alternative can be readily communicated. However, in more complex situations this method is subject to potential problems such as assuring that each alternative is evaluated in a consistent manner.

Checklist methods compare and evaluate alternatives against a specified set of criteria with no compensatory rules or tradeoffs. Typically, this involves posing a series of questions related to the individual criteria that require a yes or no response, such as:

• Is the alternative protective of human health and the environment?
• Is the alternative effective in the long term?
• Does the alternative use proven methods or technology?
• Is the estimated cost below a defined threshold?

This approach may be useful for identifying dominant alternatives for screening purposes. For example, if Alternative A is better than Alternative B in at least one respect and no worse than Alternative B in any other respect, Alternative A may be considered dominant.

Economic methods use economic procedures and principles to translate non-commensurable units into monetary units. This methodology relies largely on determining an individual's willingness-to-pay (the amount that individuals affected by the project would be willing to pay for the defined benefits), and the availability of market prices that relate to the benefits. By their nature, many of the benefits associated with environmental improvements cannot be readily determined based on market prices; however, this type of method may have application in situations where the property is being considered for sale and/or redevelopment. In this case, it may be possible to directly relate the cost of implementing an alternative to the beneficial value of land improvement.

Pairwise comparison methods use the sequential comparison of alternatives in pairs as a basis for subsequent ordering of preferences. In its simplest form, the procedure develops a measure of how frequently one alternative is superior to another based on the various evaluation criteria. This is improved using fuzzy set procedures, which is based on subjective interpolation, and is used to identify inefficient alternatives (those that are dominated by other alternatives). In this case, each alternative is numerically ranked for each evaluation criteria. Initially, two alternatives are compared to determine dominance, i.e., which of the two alternatives has the greater number of occasions of dominance. The dominant alternative is then compared to the next alternative, and so on, until one dominant alternative is identified. The method can be based on either a non-parametric or parametric ranking; however, in both cases, the assignment of the ranking values may be subjective. Also, the relative importance of each criterion is not reflected in the procedure unless the criteria are ranked into groups.

Matrix methods use a matrix for the summary, comparison and evaluation of criteria and alternatives, based on professional judgment (as an extension of ad hoc methods). In this case, weight factors are applied to each evaluation criterion to reflect its overall importance, and ranking factors are applied to each alternative (for each criterion). These are multiplied and summed to develop an overall score. In this manner, alternatives that score well can be considered to be superior to other alternatives. This method relies on subjective assignment of the weight and ranking factors, and therefore would need to be supported by the assessor's justification for assigning the factors. It is an improvement over ad hoc methods in that all evaluation criteria need to be considered for each alternative, and it is amenable to sensitivity analysis by examining the effects of changes in the factors. Both the pairwise comparison and the matrix methods are transparent in the identification of the preferred alternative and hence potentially very useful in public consultation.

Expert support tools that may assist the custodian in completing the preceding evaluation include the Guidance and Orientation for the Selection of Technologies (GOST) and the Sustainable Development Tool (SDT). GOST is a technology database that contains individual fact sheets on a host of treatment technologies/approaches. The user is prompted for a series of inputs regarding contaminant and site data (e.g., hydro-geologic conditions), which results in the identification of a number of technically feasible R/RM options. Custodians could consider the use of GOST as early as Step 5 and during Step 7 of the 10-step federal process, primarily to identify potential candidate technologies/approaches for management of their sites. A secondary benefit of GOST is that it provides assistance to the custodian in confirming the necessary data to be collected during the environmental site assessment (ESA), via the required inputs to the model, to support this evaluation.

Once the custodian has identified a suite of potential technologies/approaches using GOST, a secondary evaluation can be conducted using SDT to evaluate and compare up to five separate treatment options from the perspective of the three pillars of sustainability: economic, social and environmental. Custodians can choose from a suite of parameters for all three elements--and further, use weightings for each parameter--to reflect their specific site situation. The output from the model is both graphical and numerical, such that it serves as a communication as well as an analytical tool. This approach allows for stakeholder engagement and incorporation of multi-stakeholder requirements. The intent is that custodians will select the most balanced alternative with the cost in mind; SDT will help them to incorporate sustainability aspects into their evaluation process when identifying the preferred alternative.

Part 2: example alternatives assessment

An example of alternatives assessment using the pairwise comparison and matrix methods based on a contaminated site scenario is presented below.

Use of evaluation metrics to select the preferred remediation/risk management (R/RM) alternative

Table 2 includes the potential evaluation metrics within various categories that can be used as part of the process for identifying the preferred R/RM alternative. Although Table 2 lists a number of evaluation metrics associated with each category, not all evaluation metrics will be employed in an actual evaluation. In practice, it is only necessary to employ the evaluation metrics that are relevant to discriminating between the R/RM alternatives. Hence, only a subset of the potential evaluation metrics will be employed in any particular evaluation.

As apparent from the list of potential evaluation metrics, individual metrics are not measured using the same units, and hence they are not additive. As a result, they cannot be combined in a simple manner. Instead, a means of combining the value of an alternative must be made relative to the various evaluation metrics, to determine which of the alternatives is preferred.

An additional dimension of the evaluation criteria must also be acknowledged; if an alternative does not attain a threshold (e.g., with respect to human health and the environment), that alternative is not acceptable and is not considered beyond the first level of analysis.

Example problem definition

The following example demonstrates how the methodology is applied. Please note that this problem situation has been kept fairly simple in order to focus on the methodology rather than on precise complexities that may arise in practice. Furthermore, the exact details of the preferred strategy are not supplied but are assumed to be consistent with good practice, specifically for a remote site.

Consider the following situation:

A waste disposal pit and an underlying groundwater plume have been identified at a site. The alternatives for the remediation of the disposal pit were identified as capping the pit, or excavation and disposal of the waste.

For the underlying groundwater plume, the alternatives that will be considered are monitored natural attenuation (MNA) and groundwater treatment with MNA combined (treatment/MNA). The duration of these options will vary, since treatment will promote a more rapid reduction in contaminant concentrations. The status quo (“do nothing”) option associated with the groundwater plume should also be considered, as there may be no need to undertake action, and it will provide a baseline for comparison of this scenario.

The individual alternatives, as classified into vertical sets of options, are illustrated in Table 3. Other options may be available, for example cap and treatment/MNA, but are not included in this example in order to maintain simplicity.

These alternatives include elements related to both risk management (i.e., cap and MNA) and remediation (i.e., excavate/dispose and groundwater treatment/MNA). We are now interested in selecting the preferred alternative, where the preferences between the options regarding long-term effectiveness may be different, for example, than cost considerations.

To proceed to the next step, each of the alternatives needs to be considered with respect to each of the evaluation criteria within the categories. This step is accomplished in the following sub-tables, as follows:

(i)The alternative is judged to be unacceptable or excluded in terms of threshold levels and, hence, is no longer considered.

(ii)Table 4(a) summarizes the attributes of each of the alternatives relevant to effectiveness. It should be noted that the only relevant effectiveness evaluation criteria are the long-term effectiveness and the reduction of toxicity, mobility or volume.

(iii)Table 4(b) summarizes the attributes of each of the alternatives relevant to “implementability.” All of the alternatives involve the application of proven technologies, none are innovative, all involve the necessity to obtain permits, etc. This means that the discriminating factor between the alternatives is the time required for implementation (e.g., MNA requires a lengthy period for site remediation whereas capping is implemented relatively quickly). Impacts and risks to the environment during implementation must also be a consideration (i.e., consider the risks associated with the transport and disposal of the excavated material).

(iv)Table 4(c) describes the attributes of the various alternatives in terms of costs. In this application, the costs are determined in terms of present worth (or net present value) and therefore show the combination effect of construction costs, operation and maintenance costs, and discount rate.

(v)Table 4(d) describes the attributes for different alternatives in terms of the “Other” category. This may be a relevant consideration in selecting between the alternatives in that there is long-term liability for ensuring that MNA functions as intended, as opposed to, for example, the excavation and destruction of the wastes. For MNA, there is some degree of long-term liability associated with the site. Also, the potential impacts on future site operations may be a consideration.

The set of Tables 4(a) through 4(d) summarizes how the alternatives are measured with respect to each of the evaluation criteria. The next stage is to identify which of the alternatives is/are the preferred alternative(s). This will be accomplished using the two separate procedures designed for this identification, namely (i) the pairwise comparison method and (ii) matrix weighting procedures.

Identification of preferred alternative

Using the pairwise comparison method

Alternative 2 has four evaluation criteria in which it is preferred to Alternative 1, and there are three criteria in which Alternative 1 is preferred to Alternative 2. In this situation, Alternative 2 moves on to be compared with Alternative 3. It is noted that this comparison suggests that there is little difference between Alternatives 1 and 2.

In a more complete assessment of impacts, other considerations such as off-site impacts like transportation of excavated material and liability/risk associated with disposal might also be evaluated with regard to each alternative. Please note this type of evaluation does not give weight to the evaluation criteria; it only allows a preference for one alternative method or another. A weighted matrix example is explained later in this appendix.

In this comparison, Alternative 3 is preferred to Alternative 2 with regard to five evaluation criteria, whereas Alternative 2 is preferred to Alternative 3 in only one criterion. This indicates that Alternative 3 is the preferred alternative remediation option.

Based on the above, it could be concluded that Alternative 3 is the preferred alternative, if all evaluation criteria were considered to have equal weight (or importance), as is the case with this method. Weightings are applied in the matrix method discussed below.

Using matrix weighting procedures

Two sets of weighting factors are required:

• The factor weights for the evaluation criteria within each category, where the sum of the factor weights equals one. For example, within the effectiveness category there are two evaluation criteria (long-term effectiveness and reduction of toxicity, mobility or volume), each of which is assigned a factor weight.
• The priority group weights, to reflect the relative importance of each category and assign values such that the sum of the priority group weights equals one. In this case, each of the four categories (effectiveness, implementability, cost, other) is assigned a priority group weight.

The selection of the weighting factors needs to consider the viewpoints of the interested parties, recognizing that different stakeholders may be more sensitive to specific evaluation criteria than others. However, the procedure does allow sensitivity testing to determine differences in the analysis resulting from changes in the weight factors.

Ranking of one alternative relative to another

In the example matrix, each of the alternatives is ranked relative to the others using non-parametric means, such that the best of the three alternatives associated with each of the evaluation criteria receives a 3, the second-best gets a 2, and the third-best gets a 1. In the event of a tie, the average of the two is assigned to both.

Simple matrix weighting calculations are summarized in the table below, which shows that Alternative 3 is the preferred alternative of the three (i.e., has the highest score).

Ranking of each alternative on a scale of one to ten

Another option is to rank the values on a scale from one to ten using parametric means. This allows the assessor to determine, for example, the magnitude of the differences between the alternatives for individual evaluation criteria.

Additional considerations

The example problem was kept very simple to allow the primary focus to be on the selection procedure for the preferred alternative. However, it should be clear that the process may be considerably more complex in a real situation. Examples of the challenges that could arise include the following:

• There may be more than one evaluation criteria necessary to discriminate between the preferred alternatives in a particular application. For example, there could be both long- and short-term differences in the effectiveness of different alternatives. In this situation, and if both long- and short-term ramifications are better for Alternative A in comparison with B, the approach is relatively simple in that both could be combined into a single metric by which the alternatives can be compared. The challenge will be where Alternative A is better than B with respect to short-term effectiveness, and B is better than A with respect to long-term effectiveness. In this situation, it may be necessary to employ the preferred alternative within an individual category first, and then proceed to the next level of assessment.
• The procedures are readily transparent and are apparent to reviewers. Hence, discussion on the assignments can be focused on points of controversy, should they exist.

The procedures are straightforward to apply and test the sensitivity of the selection by allowing different methods to arrive at the same conclusion.

There is merit in completing evaluations using one or more procedures, for example pairwise comparison or matrix weighting comparisons; if the results are the same, it demonstrates that the findings are robust.