Metal mining technical guidance: revised guidance for sample sorting, annex

Annex: Review of Cycle 2 Benthic Processing Protocols and QA/QC Issues

• Introduction
• Sorting Efficiency
• Subsampling Methods
• Subsampling Precision and Accuracy

Introduction

The technical guidance documents prepared for both the pulp and paper and metal mining EEM programs have provided guidance for laboratory sample processing, subsampling protocols and quality assurance. However, detailed protocols and acceptability criteria for subsampling have been lacking. Quality of laboratory benthic sorting can affect the accuracy of the benthic invertebrate community endpoints required to assess effects on fish habitat (i.e., total density, taxa richness, Simpson's Diversity and Bray-Curtis). If protocols within a single survey are consistent, minor errors in sorting efficiency are less problematic with regard to data interpretation of effects at a particular mill/mine. However, there may be a significant source of error in interpretation for individual mills/mines or for analyses performed at a larger scale (i.e., national or regional) if sample processing errors occur between samples, areas or mills.

To identify the scale of potential processing or subsampling issues the first step undertaken involved a detailed review of the methods reported in the Cycle 2 EEM P&P interpretative reports. The information extracted from these reports included: the number of studies where laboratory subsampling was performed, the type of subsampling, if detailed protocols were provided, if density correction factors were reported and calculated appropriately, if subsampling error (precision and/or accuracy) was reported and calculated appropriately and finally if QA/QC criteria for sorting and subsampling were established and adhered to in these studies. The intent of this review was not to re-invent established protocols but to review the level of detail in the current guidance documents and establish if the resulting interpretative reports provide the sufficient information to allow assessment of data quality. Areas where more specific guidance may be beneficial to the EEM program were identified, resulting in a revised draft of the guidance for benthic invertebrate sorting protocols, specifically pertaining to Sections 6.2.2 in the PP TGD and 5.22 in the MM GD. The following review covers three key issues reviewed from the Cycle 2 Interpretative Reports; sorting efficiency, subsampling methods, and subsampling precision / accuracy calculations.

Sorting Efficiency

The basic processing of benthic invertebrate samples involves the time-consuming removal of organisms from large amounts of debris. Inevitably, processing errors occur during this sorting phase regardless of how diligent the processor. The first QA/QC component of benthic invertebrate sample processing is the requirement to assess sorting efficiency (i.e., the proportion of total organisms extracted from the sample upon sorting). High sorting efficiencies will ensure that endpoint calculations are reasonably reliable and without bias between samples. The recommendation for assessing sorting efficiency for the PP and MM EEM programs is that at least 10% of all samples from each study be resorted and any organisms found on the second sort be enumerated. The criteria for an acceptable sort is that $ 90% of the total number of organisms are recovered during the initial sort. In a review of the Cycle 2 Interpretative Reports 98% of all studies reviewed (87 mills) reported that this sorting QA/QC procedure was followed. However, only 58% of these mills reported the actual efficiency attained on the samples for a particular study. Of those reporting the efficiency, all attained the $ 90% target and most (75%) achieved $ 95% efficiency. Assuming, that the lack of reporting study-specific sorting efficiencies, does not represent efficiencies lower than the target, this aspect of benthic sorting QA/QC appears to be well understood and applied. However, sorting efficiencies should be reported for all studies in the standardized tables provided in the revised guidance.

Subsampling Methods

If the processing protocol involves subsampling of the sample, the second component of QA/QC involves an evaluation of subsampling error. As with basic sorting of benthic samples some level of error will result with any subsampling method. The error of concern for subsampling methods is how accurately the method estimates the total number of organisms in the entire sample. The similarity between two subsamples (i.e., the precision) is of less importance than the accuracy of the estimate. This said, if all subsamples were processed from a sample and the precision was high across all subsamples, the accuracy of each will also be high. However, if precision is low or variable, accuracy will vary with subsample. Thus, both of these measures of subsampling error should be evaluated with an appropriate QA/QC program.

To evaluate subsampling protocols in use for the EEM program, 83 mill studies were reviewed in detail. Close to 90% of all studies reviewed (83 mills) used some type of subsampler to assist in the processing of benthic invertebrate samples (Table 1). Although, subsampling methods vary, the simple objective is to divide up a large sample, into several portions, each of which are representative of the entire sample. These smaller portions are processed more efficiently and cost-effectively while producing reliable estimates of the total number of organisms in the sample. The three most common methods divide up the samples based on area, weight or volume of the sample and are appropriate for a range of sample types. The majority of Cycle 2 studies used an area based subsampling method. Although, there are a variety of area based techniques, all estimate the total number of organisms in a sample by the aerial proportion extracted off a planar surface. The other commonly used methods involved dividing the sample by weight or by volume. However, fully, 23 % of the studies did not indicate which method was used even though subsampling was used to assist in sample sorting. Several other techniques (Table 1, other category) were briefly described but their effectiveness is uncertain as they were not documented by literature evaluations. Finally, the fixed count method, a subsampling method used widely in the US for field sampling programs, was not used in any of the EEM reports reviewed.

Table 1: Types of subsamplers used in the 83 Cycle 2 mill studies reviewed

Subsampling Precision and Accuracy

Regardless of the subsampling technique used, documentation of the accuracy of the estimate is essential to ensure that the data is comparable within and between studies. In fact, the main criteria to evaluate a subsampling technique is an evaluation of it's ability to accurately estimate the numbers and types of organisms in a sample. The reporting of subsampling error was reviewed from 57 mill studies from Cycle 2 P &P. Unfortunately over half of these studies (56%) did not report the error associated with the subsampling technique used. Of those that reported error, the majority (72%) reported the precision obtained when comparing two subsamples. For example;

1. a count in subsample A = 289
2. a count in subsample B = 316
3. the reported precision between theses two subsamples would be 8.5%
4. (1-(89/316))x100.

If all the subsamples from this particular sample had similar precision, then the accuracy will also be close to 9%. However, without sorting the remainder of the sample, accuracy cannot be determined. The studies which did report the accuracy of the subsamples, did so as suggested in the guidance documents. For 10% of all samples several subsamples were sorted and then the remainder of the sample was sorted entirely. The subsampling error was calculated by comparing the estimates from the subsamples to the actual count. For example:

1. a count in subsample A = 289, representing 15% of the sample by volume, for an estimate of the total in the sample of 1927
2. a count in subsample B = 316, representing 15% of the sample by volume, for an estimate of the total in the sample of 2106
3. the count in the remainder of the sample = 1359, for a actual total of 1964
4. the reported precision would be the same as in the first example, 8.5 %
5. the reported accuracy would be -1.9% and +7.2% for sample A and B respectively.

This type of QA/QC information is essential to ensure that the subsampling program is accurately estimating numbers of organisms in the sample. A final note on the Cycle 2 Interpretative Reports is that, the correction factors used to estimate the total number of organisms in a sample were generally not reported. This is a simple calculation, especially for subsampling devices which simply divide the sample in half, but should, nevertheless, be reported.

In addition to reviewing the Cycle 2 interpretative reports, the guidance documents for both PP and MM were reviewed to assess whether further guidance may assist in obtaining more consistent reporting of QA/QC results. As there appears to be considerable variation in how error is calculated and reported a redrafting of the guidance pertaining to sections 6.2.2 in the PP and 5.22 in the MM guidance documents is underway. Included is a review and detailed description of subsampling techniques and suggested reporting procedures for QA/QC. An important addition to this guidance is a specific recommendation of an acceptable level of error for laboratory subsampling which was lacking in previous versions. Perhaps due to the lack of this criteria, none of the Cycle 2 studies which reported error (either precision or accuracy) outlined corrective action if error levels were unacceptable. The overriding objective of subsampling is to reduce the substantial effort involved in processing benthic samples but not at the cost of data quality. For the basic sorting efficiency, the acceptable error was set at 10%. For subsampling error the majority of Cycle 2 studies applied a 20% precision rule. That is, if the precision between two subsamples was < 20% the error was acceptable (see example on precision above). Although this level of precision was suggested in the guidance documents in terms of determining how many grab samples (i.e., field subsamples) should be taken at a given station, it was not explicitly suggested as an acceptable level for laboratory subsampling. An approach to setting these acceptability rules and a follow up course of action if the subsampling protocols fail these rules is included in the expanded guidance.