Monitoring Protocols

Part 2, Monitoring Protocols, provides Forest Service personnel detailed instructions for compiling, analyzing, and interpreting the data and resulting trends for all 28 measures described in this technical guide. This section provides an overview of how to implement wilderness character monitoring (WCM) and describes the standard sections that are included for each measure. The remaining sections 2–6 provide detailed protocols, that is, the step-by-step instructions for all measures, organized by their respective qualities:

2.0—Untrammeled Quality

3.0—Natural Quality

4.0—Undeveloped Quality

5.0—Solitude or Primitive and Unconfined Recreation Quality

6.0—Other Features of Value Quality

The measures described in sections 2 through 6 use the same section numbering that appears in part 1 of this technical guide to allow users to crosswalk between the measure descriptions in part 1 and the detailed monitoring protocols in part 2. Hyperlinks were created to provide easy access between these two parts. The glossary and references for both part 1 and part 2 are included at the end of the main text in part 2.

In addition, the following appendices are included at the end of part 2:

Appendix 1—Summary of Key Implementation Attributes for All the Measures in Each Quality

Appendix 2—Measures Considered but Not Used

1.1 How to Implement Wilderness Character Monitoring

There are three basic steps to implementing Forest Service WCM: (1) select measures, (2) review roles and responsibilities, and (3) begin data compilation, analysis, and entry.

Step 1: Select measures. The local unit that has the lead for each wilderness selects measures. The approach for selecting measures in WCM is similar to that used in Wilderness Stewardship Performance (WSP). There are five types of measures.

1. Required—The measure is required for all wildernesses.
2. Required to Select at Least One—At least one measure must be selected from the set of several potential measures; selections should be based on relevance to a wilderness, data availability, and additional measures from the set may also be selected if relevant.
3. Required if Relevant—If a wilderness uses the Other Features of Value Quality, one or more of these measures are required to be selected.
4. Optional—The measure may be selected if relevant to a wilderness.
5. Locally Developed Measures—In addition to the measures identified in this technical guide, the local unit may develop new measures for other attributes considered integral to wilderness character for the individual wilderness.

Required and required if relevant measures were developed to ensure a level of national consistency and cannot be modified by local units. Likewise, the primary measure selected from a set of "required to select at least one" measures also cannot be locally modified. If local units are interested in making a small change to a required measure's protocol (e.g., if a unit wants to include unique developments that are not encompassed by the monitoring protocol for the measure Index of Authorized Non-Recreational Physical Development, such as radio collars or large trash objects), they should contact their Regional Wilderness Program Manager and their Wilderness Information Management Steering Team representative to discuss the appropriateness and feasibility of this change. Any substantial changes to the protocols described in this technical guide will occur through the change management process (see section 1.8 in part 1).

In contrast to the required measures, optional measures may be adapted as necessary to suit local units. If multiple measures are selected from a set of "required to select at least one" measures, the additional measures may also be locally adapted (although at least one measure from the set must remain unmodified). For example, local units may want to track fire suppression through the optional measure Percent of Emergency Incidents Using Motor Vehicles, Motorized Equipment, or Mechanical Transport, but lack data on the other types of emergency incidents included in the monitoring protocol; in this case, they could modify the measure to quantify only the percentage of emergency fire suppression incidents using motor vehicles, motorized equipment, or mechanical transport. Similarly, if local units select both Index of Recreation Sites Within Primary Use Areas and Miles of Unauthorized Trails from the same set of "required to select at least one" measures, they could adjust Miles of Unauthorized Trails to track only miles of unauthorized outfitter and guide trails (but would then have to leave the measure Index of Recreation Sites Within Primary Use Areas as is).

Locally developed measures are expected to be rare, but may be included if (1) an attribute of wilderness character that is not included in this technical guide is integral to the area's wilderness character and is vulnerable to human-caused degradation, and (2) the local unit can reliably monitor that element into the future with sufficient data adequacy. If a locally developed measure is being considered, the local unit must contact their Regional Wilderness Program Manager and their Wilderness Information Management Steering Team regional representative to discuss the appropriateness and feasibility of the proposed measure. A locally developed measure can never replace a required measure. For example, if a resource specialist knows about better or more appropriate local data than what is included for a required measure in this technical guide, for national consistency the required measure must still be used and the better or more appropriate local data could then be used as the basis for an additional locally developed measure.

See section 1.5.3 in part 1 of this technical guide for general information on measures.

Step 2: Review roles and responsibilities. Local units review the roles and responsibilities for data compilation, analysis, and entry for each measure. The protocol for every measure in part 2, sections 2–6, begins with a table that explicitly summarizes the local and national tasks for that measure, followed by step-by-step instructions. Appendix 1 describes the local and national tasks for all measures, as well as other key implementation attributes. Definitions of the terms related to data compilation, analysis, and entry and how they are used in part 2 of this technical guide are described below:

• Data compilation—Refers to acquiring data for use in WCM. This includes collecting or gathering data from the field (e.g., counting the number of administrative installations) or retrieving existing data from Natural Resource Manager (NRM) or other local or external sources (e.g., acquiring state data on the spread of aquatic invasive species). This may also include compiling legacy data if appropriate and available for a measure (see section 1.5.3 in part 1 for additional information on data sources and legacy data).
• Data analysis—Refers to actions taken to manipulate data to derive a single value for the measure. This includes processing data retrieved from NRM (e.g., deriving the average Animal Unit Months [AUMs] across all wilderness allotments), calculating an index value (e.g., multiplying the trail distance by the trail class), or analyzing spatial data (e.g., performing a GIS analysis of wilderness acreage away from internal developments). Throughout part 2, average and mean are used interchangeably to describe the central tendency of the data.
• Data entry—Refers to entering the data into the appropriate NRM database, local database, or the Wilderness Character Monitoring Database (WCMD). While not all measures require data to be entered in NRM or a local database, all measures require data to be entered in the WCMD. Guidance for accessing and using the WCMD will be released now that WCM was formally implemented in the Forest Service in 2018 (see section 1.7.3 in part 1).

Some measures, such as Number of Authorized Actions and Persistent Structures That Manipulate Plants, Animals, Pathogens, Soil, Water, or Fire, require data compilation, analysis, and data entry in the WCMD to be completed by local units. For other measures, such as Concentration of Ambient Ozone and Deposition of Nitrogen, a central data analyst at the national level compiles, analyzes, and enters data in the WCMD. Finally, for a few measures, such as Extent of Waterbodies with Impaired Water Quality, local units and a central data analyst must work together to complete data compilation, analysis, and data entry in the WCMD. In some cases, the data analysis for a measure may be performed automatically by the WCMD or by NRM-WCM. Data analyses by NRM-WCM may also automatically retrieve data from other NRM applications (e.g., data entered annually in NRM-Wilderness may be retrieved by NRM-WCM to calculate a measure value). As NRM is currently undergoing extensive revisions, for certain measures it is unknown at this time whether data analyses will be performed by NRM or whether users will need to do them by hand; therefore, instructions are included in part 2 for all measures even though local users may not need to perform these calculations in the future. Similarly, for certain measures the location of NRM data for compilation (including the specific NRM application that data are stored in and which attributes are relevant for a measure) may change in the future; the WCM Program Manager will be responsible for tracking and updating changes to NRM data compilation protocols.

Step 3: Begin data compilation, analysis, and entry. Local units and the central data analyst begin data compilation, analysis, and entry for the selected measures using the standard procedures described in part 2, sections 2–6. For all measures in these sections, detailed instructions describe the logical steps for data compilation, analysis, and entry that either a central data analyst or local user would follow. The instructions in these sections were developed to be at the appropriate geographical scale (either national or local) and to minimize the time required to gather the information. The measure value reported for each measure should be rounded to the nearest whole number (i.e., values from 1.1 to 1.4 become 1, and values from 1.5 to 1.9 become 2) unless stated otherwise in the protocol for a measure.

1.2 Standard Implementation Sections Described for Each Measure

For every measure included in sections 2–6, the following sections, in order, provide guidance for compiling, analyzing, and entering the data into the WCMD, as well as for determining and interpreting the trend in these measures.

1.2.1 Protocol

The protocol provides step-by-step instructions on how to compile, analyze, and enter the data necessary to determine the trend in the measure. Each protocol produces a single value for each measure (the measure value) that is used to derive the trend in this measure (see section 1.5.3 in part 1 for definitions and procedures on data handling). Protocols described in the sections 2–6 are based on the best available scientific information for monitoring wilderness character and comply with requirements of the 2012 Planning Rule (36 CFR 219) and the Data Quality Act (P.L. 100–554).

While most measures have a single set of instructions for the protocol, a few measures have several "protocol options." These protocol options take into account differences among wildernesses in data sources, data availability, and data adequacy, and allow local users to select the most appropriate protocol option for their unit. For example, the measure Index of Encounters has several protocol options based on the type of data currently being collected for a wilderness: camp encounter data, traveling encounter data, visitation data, or no data collection. When multiple protocol options are described for a measure, local units must select which one they will follow to compile, analyze, and enter the data. If more than one protocol option is relevant and feasible for a local unit to monitor, the unit may include additional protocol options as locally developed measures. For example, if a wilderness collects data on both campsite encounters and total visitation, they could use the encounter data as the selected protocol option for the required Index of Encounters measure and include the visitation data as an additional locally developed measure.

If better data sources become available, it may be appropriate to change to a different protocol option. The decision to change protocol options must be weighed carefully as it may alter the trend in the measure. When local units consider making such a change, they should contact their Regional Wilderness Program Manager and their Wilderness Information Steering Team regional representative to discuss the appropriateness and feasibility of this change. When protocol options are changed, it is important to document when the change occurred, the reason(s) for this action, and the potential impact on interpreting trend in wilderness character. Sections 1.5.3 and 1.8 in part 1 provide information on making changes to data sources and measures.

1.2.2 Caveats and Cautions

For each measure, caveats and cautions related to use of the protocol are described. This section may expand on concerns about the availability or quality of data or provide additional information about assessing the trend in the measure. For example, caveats may include availability and variability of data by geographic region, concerns about the locations of monitoring sites, and pending changes in databases or data sources.

1.2.3 Data Adequacy

Data adequacy is the reliability of the data to assess trends in the measure. It encompasses both data quality and data quantity (described below). Each measure included in this technical guide contains an evaluation and discussion of data adequacy. The data adequacy rating is based on a broad national assessment of existing databases and other sources of information about a measure. For each measure, local units must validate the general determinations of data adequacy that appear in this technical guide for their specific wilderness. Data adequacy is not used in determining the trend in a measure, but it is crucial information for interpreting this trend (e.g., if there is a degrading trend but data adequacy is low, then confidence in this trend would also be low) and for revealing if more effort is needed to collect more or better data to improve confidence in the resulting trend.

Each local unit is required to use the best available scientific information for all selected measures. In some cases, older legacy data (e.g., a plant survey or encounter monitoring conducted 10 or 20 years ago) may be all that is available; in these cases local resource specialists need to carefully scrutinize these data to see if they are still valid or appropriate to use in WCM. If such data are used, data adequacy also needs to be carefully evaluated. When measures have multiple potential data sources, data adequacy helps determine which sources are most appropriate to use for an individual wilderness. In addition, some measures incorporate multiple sources of data to produce a single measure value. Sections 2–6 provide an assessment of data adequacy for each data source, but do not integrate those evaluations into a single overall determination of data adequacy for the measure if multiple data sources are combined. Each local unit must determine the overall data adequacy of these types of measures on a case-by-case basis.

Data quantity refers to the level of confidence that all appropriate data records have been gathered. In determining the best available scientific information for a local unit, "available" refers to information that currently exists in a useful form, and that does not require further data collection, modification, or validation. If the available data are insufficient in quantity, they may still be considered the best available scientific information for the local unit. Data quantity is described by the following three categories:

1. Complete—This category indicates a high degree of confidence that all data records have been gathered. For example, to assess the occurrence of nonindigenous plants, a complete inventory of a wilderness was conducted or all likely sites were visited. Similarly, to assess encounters, all trailheads were inventoried.
2. Partial—This category indicates a medium degree of confidence that all data records have been gathered. Some data are available but are generally considered incomplete, such as with sampling. For example, to assess the occurrence of nonindigenous plants, only a partial inventory was conducted; to assess encounters, only selected trailheads were sampled.
3. Insufficient—This category indicates a low degree of confidence that all records have been gathered. Few or no data records are available. For example, no inventory for nonindigenous plants has been conducted, and encounters were not assessed anywhere, requiring professional judgment in both cases.

Data quality refers to the level of confidence about the data source and whether the data are of sufficient quality to reliably identify trends in the measure. Data quality is assessed by the data's accuracy (the degree to which the data express the true condition of the measure and not other sources of variation affecting the measure), reliability (the degree to which the data follow established or well-developed scientific protocols), and relevance (the degree to which the data are spatially and temporally appropriate for the measure). In general, the highest quality data will be considered the best available scientific information. Data quality is described by the following three categories:

1. Good—This category indicates a high degree of confidence that the quality of the data can reliably assess trends in the measure. Data are highly accurate, reliable, and relevant for the measure. For example, data on the occurrence of nonindigenous plants are from ground-based inventories conducted by qualified personnel; for encounters, data comes from encounter monitoring following the national minimum solitude monitoring protocol.
2. Moderate—This category indicates a medium degree of confidence about the quality of the data. Data are only moderately accurate, reliable, or relevant. For example, data on nonindigenous plants could come from national or regional databases; for encounters, data could come from visitor permit data.
3. Poor—This category indicates a low degree of confidence about the quality of the data. The accuracy, reliability, or relevancy of the data is minimal or unknown. For example, data on nonindigenous plants and encounters data could come from professional judgment.

Local resource specialists must evaluate data quantity and quality for all potential data sources.

An overall determination of data adequacy is derived by combining the assessments of both data quality and quantity (see table 2.1.1) and is categorized as high, medium, or low.

Table 2.1.1—Data adequacy matrix displaying data quantity and data quality to determine data adequacy.

		Data quality
		Good	Moderate	Poor
Data quantity	Complete	High	Medium	Medium
	Partial	High	Medium	Low
	Insufficient	Medium	Low	Low

1.2.4 Frequency

Frequency is how often data are compiled, analyzed, and entered into the WCMD. Some measures only need data compilation, analysis, and entry at 5-year intervals because the data are unlikely to change during this period (e.g., the number of dams or communication installations in the measure Index of Authorized Non-Recreational Physical Development). Other measures, however, will require annual data compilation, analysis, and entry because the data are likely to change from year to year (e.g., the number of administrative uses of motorized equipment in the measure Index of Administrative Authorizations to use Motor Vehicles, Motorized Equipment, or Mechanical Transport). Units may compile, analyze, and enter data for a measure at more frequent intervals than required for each measure, but may not compile, analyze, and enter data at less frequent intervals.

1.2.5 Threshold for Change

For each measure, a threshold, or the amount of change in the data necessary to qualify as a meaningful change in the measure, is identified. This threshold varies across measures due to (1) how inherently variable the data for the measure are likely to be from one year to the next, and (2) the adequacy of the data. Three standard categories are used for thresholds:

1. Any change
2. Percent change
3. Statistically significant change based on regression

The any-change threshold applies to measures for which any change in the data would be meaningful from the perspective of wilderness character. This threshold is typically used for measures where change over time is unlikely or where there is high certainty about changes in the data. For example, this threshold applies to the measure Acres of Inholdings because change over time is relatively infrequent and any increase or decrease in inholding acres would be a meaningful change. The any-change threshold is also used for measures where categories are used to determine the trend in the measure. For example, when professional judgment is used for the measure Acres of Nonindigenous Plant Species, the any-change threshold is used because any change between the defined categories (none, low-, moderate-, or high-estimated percentage of a wilderness occupied by nonindigenous plants) would be, by definition, a meaningful change.

The percent-change threshold applies to measures that are less sensitive to change, that show variation from year to year, or that have medium data adequacy. Three types of percent-change thresholds are assigned to measures: 3-percent change, 5-percent change, and 10-percent change. The larger percentages indicate a higher likelihood of annual variation or a lower expected data adequacy. For example, the measure Index of Encounters uses a 10-percent change threshold due to the natural variation in visitation from year to year and the high likelihood of a low sampling intensity. Similarly, the measure Index of Nonindigenous Terrestrial Animal Species uses a 5-percent threshold because species distributions vary naturally over time and data adequacy is likely to be low for many wildernesses. Finally, the measure Index of Non-Recreational Physical Development uses a 3-percent change threshold to screen out minor or insignificant changes in the number of developments, such as those discovered when validating data from NRM or from a minor extension of a grazing fence line.

The regression threshold applies to certain measures once they have accumulated a sufficient amount of data and is used because it provides statistical rigor in the long-term analysis of trends in the measures. Regression is a commonly used and relatively simple statistical technique to determine if there is a significant change in one variable, for example, the amount of nitrogen deposition or the number of trammeling actions, in relation to another variable, such as time over several years. There are many different regression models (that is, types or forms of regression) and the appropriate model will be chosen by the central data analyst in consultation with a statistician based on the properties of the data used for each measure.

Using regression to determine whether a trend is statistically significant requires the user to assign the desired degree of confidence, or certainty, in the results, called the alpha level (there is an extensive literature on this topic that is beyond the scope of this technical guide). For all measures that use regression in this technical guide, an alpha level of 0.1 will be used in determining statistical significance, meaning that there is a 10-percent chance of concluding that there is a significant trend when in reality there is not a trend, or conversely that there is 90-percent confidence or certainty that the trend is real. This alpha level allows an appropriate balance between the need to catch trends early while maintaining as much statistical rigor as possible in correctly identifying meaningful trends (see appendix B in Landres et al. 2009 for details on the selection of this alpha level and use of regression).

Typically, at least five data values are needed when using regression, however other factors need to be considered and the central data analyst will need to consult with a statistician to ensure that the data are sufficient and appropriate for using regression. For measures that use an annual frequency (e.g., Index of Administrative Authorizations to Use Motor Vehicles, Motorized Equipment, or Mechanical Transport), regression can be used after 5 years of data compilation (measures that use a 3-year rolling average will need 7 years of data compilation) and the trend that is reported for the 5-year WCM cycle will be based on the results of the regression analysis. For measures that use a 5-year frequency (e.g., Index of Authorized Non-Recreational Physical Development), regression can be used after 20 years of data compilation. When regression is used, all the available and appropriate data, including legacy data, will be used in the analysis. The WCMD will automatically perform the regression analysis to calculate the trend for the measure, and local and national staff will not need to conduct this analysis.

Several measures start with a percent-change threshold and then switch to using regression once there are sufficient data, which is typically five measure values. Switching to regression is generally not appropriate for measures that use the any-change threshold and measures that use categories for the measure value (such as when professional judgment is used for the measure Acres of Nonindigenous Plant Species as described above). Switching from a rule-based, percent-change threshold to regression may change the resulting trend because the rule-based method determines trend by comparing the most recent measure value with the measure baseline value, whereas regression determines trend using all of the available data. Even if switching to regression causes a change in the trend from one 5-year monitoring cycle to the next, this change is appropriate because of the greater statistical rigor in using regression.

For measures that use the any-change and percent-change thresholds, trend is determined generally by comparing the most recent measure value with the measure baseline value. The WCMD will automatically calculate trends in the measures based on the thresholds for meaningful change described in sections 2–6; neither local nor national staff will need to calculate trends. However, wilderness staff interested in understanding the effects of recent administrative actions, or for other reasons, may choose to assess short-term trends by comparing the two most recent measure values even though these short-term trends are not required for Forest Service WCM upward reporting.