Volunteer Accuracy in a Benthic Macroinvertebrate Participatory Science Project

Abstract

Concerns about the accuracy of volunteer-derived aquatic macroinvertebrate identifications and the resulting influence on calculated water quality metrics can limit use of volunteer-generated data in freshwater-focused participatory science programs. To address these concerns, we assessed 357 benthic macroinvertebrate quality control (QC) samples collected by volunteers using leaf packs, kick nets, and visual assessments between 2011 and 2016 for the Environmental Quality Institute (EQI) in North Carolina, USA. We reviewed each sample for quality, and compared macroinvertebrate identifications and water quality metric scores determined by volunteers to identification and metrics determined by an entomologist. About 80% of the QC samples were identified to be of high quality, indicating proper preparation, preservation, and labeling of samples. The majority of samples that received low quality ratings were improperly or poorly preserved. We recommend use of 95% (rather than 70%) ethanol for macroinvertebrate preservation, and increased communication to volunteers during QC sampling periods to enhance their success in properly preserving samples. We observed significant (p < 0.005) linear relationships between volunteer and entomologist-derived water quality metrics including a biotic index, Taxa Richness, and percent intolerant Ephemeroptera, Plecoptera and Trichoptera (EPT) taxa. However, visual assessments – those in which no sampling equipment was used to collect aquatic macroinvertebrates – reduced the goodness of fit between volunteer and entomologist-derived biotic index scores and artificially increased the goodness of fit between volunteer and entomologist derived Taxa Richness and percent EPT scores. We recommend calculating water quality metrics based only on leaf pack and kick net samples collected by volunteers.

Introduction

In biomonitoring, living organisms are used to assess the health of the environment (Oertel and Salanki 2003). In streams, while instantaneous chemical monitoring can provide information about a specific set of parameters at one point in time, it does not effectively capture cumulative impacts on stream conditions caused by varied influences over an extended period (Li, Zheng, and Liu 2010). Government agencies responsible for reporting water quality have added biomonitoring to chemical monitoring to provide a time-integrated sample and more direct quantification of aquatic ecosystem health.

Benthic macroinvertebrates are ideal to use to assess ecosystem health because they are generally simple to collect, and they spend a significant portion of their lives under water (Wallace and Webster 1996). Unlike fish surveys that may require larger or more technical equipment, an effective benthic macroinvertebrate collection requires relatively low-tech methods (Cook 1976). In addition, benthic macroinvertebrates represent a large fraction of the taxonomic families that compose overall diversity in aquatic freshwater systems worldwide. These species inhabit a variety of stream and river types and make use of the heterogeneous nature of stream beds and stream banks. They occupy a broad spectrum of ecological niches within the stream channel, from predators to decomposers (Poff et al. 2006). Macroinvertebrates also display a variety of levels of tolerance to various physical conditions and to pollutants that affect properties such as dissolved oxygen concentration and pH (Cook 1976; Smith, Brandt, and Christensen 2011).

Water quality in North Carolina is monitored by the North Carolina Department of Environmental Quality’s Division of Water Resources (NC DEQ). The NC DEQ is responsible for implementing federal policies that dictate recreational water quality requirements. This includes evaluating water quality by using the biological communities of those water bodies as indicators. In wadable streams, NC DEQ employees use kick net, sweep net, leaf pack, log wash, and visual surveys to collect insects and other stream invertebrates and identify them to species level (NC DEQ 2016). Due to limited staff and an ever-shrinking budget, NC DEQ sampling events occur with less frequency and at fewer locations each year (Ross 2020). As in many places across the United States and around the world, participatory science programs have been implemented to supplement regulatory agency data.

In North Carolina, the Environmental Quality Institute (EQI), initiated in 1988, offers an additional source of analytical data to fill gaps in water quality assessments created by chronic underfunding in the state regulatory bodies (Maas, Kucken, and Gregutt 1991; Postelle 2009). Since 1990, the EQI has trained volunteers to collect stream and lake water samples at more than 200 stream sites monthly through its Volunteer Water Information Network program. These samples are assessed for a variety of chemical water quality parameters at the EQI’s state certified lab. In addition, since 2005, the EQI volunteers have monitored benthic macroinvertebrates at up to 49 stream sites twice per year under their Stream Monitoring Information Exchange (SMIE) project. These biological data from trained volunteers are used by the EQI employees to calculate multiple common indices describing stream health, including an SMIE Biotic Index, Taxa Richness, and number of intolerant taxa (Supplemental File 1: EQI White Paper). NC DEQ has specifically assigned three data tiers with appropriate data usage based on quality assurance protocols. SMIE data are in the Tier 2 category, which are used for basin planning, research, effectiveness monitoring, and targeting of management actions (NC DEQ n.d.).

However, stream health assessments derived from the EQI’s volunteer biomonitoring protocols are subject to a variety of quality concerns common to macroinvertebrate biomonitoring methods. These include questions about the efficacy and comparability of sampling and sorting procedures (Cao and Hawkins 2011; Feeley et al. 2012), the veracity of macroinvertebrate identifications (Haase et al. 2006) and potential influence of these factors on resulting stream health assessments (Guareschi, Laini, and Sánchez-Montoya 2017). In studies that compared professionals’ biomonitoring results to other professionals’ results, implementing standardized methods (Haase et al. 2006) and strong quality assurance measures were recommended (Stribling et al. 2008; Guareschi, Laini, and Sánchez-Montoya 2017) to minimize differences in stream quality assessments across individuals. Addressing the concern about comparability of sampling procedures, the EQI SMIE methods were modeled after those used by the NC DEQ (2016). The EQI SMIE methods were chosen for their concurrency with regulatory sampling methods, their ease of use by non-professionals, and because they incorporated assessment of numerous instream habitats. However, the veracity of macroinvertebrate identifications made by volunteers with the EQI SMIE protocol warrants further study.

At least 11 studies have assessed veracity and/or comparability of volunteer methods to professional macroinvertebrate biomonitoring methods. At the ecological level of assessment, results commonly demonstrated a high percentage of agreement or correlation (Engel and Voshell 2002; O’Leary et al. 2004; Storey and Wright-Stow 2017). This led researchers to conclude that, when properly trained, volunteers using simplified methodologies could effectively supplement professional biomonitoring (Fore, Paulsen, and O’Laughlin 2001; Gowan et al. 2007; Moffett and Neale 2015; Storey and Wright-Stow 2017; Peeters et al. 2022). When volunteers were not trained, sorting was biased towards larger specimens and identification errors occurred (Nerbonne and Vondracek 2003). Even when trained, volunteers sometimes made identification and subsampling mistakes (Pinto et al. 2020). For instance, volunteers using a qualitative method under-sampled tiny immobile taxa as compared to researchers using a quantitative method to collect macroinvertebrates (Krabbenhoft and Kashian 2020). Further, volunteers collected significantly fewer taxa and calculated less total abundance of macroinvertebrates per sample than the researchers (Krabbenhoft and Kashian 2020; Peeters et al. 2022), and overestimated stream quality at sites by oversampling certain habitats (Nerbonne et al. 2008) and sensitive taxa (Storey and Wright-Stow 2017; Peeters et al. 2022). Validation of methods (Engel and Voshell 2002) and development of a quality assurance project plan (Peeters et al. 2022) were recommended as mechanisms to minimize errors in sampling, sorting, and identification.

Accordingly, since 2011, to assess veracity of the EQI SMIE project results, quality control (QC) samples have been collected at a subset of sampling locations. The EQI staff calculate percent similarities between the volunteer and entomologist identifications and counts. These data are used to provide feedback to individual volunteers after every field season. Volunteers are provided guidance about specific taxa to review and potential improvements they can make to their sampling procedures. Volunteers are required to be retrained if their score is less than 85% similar to the entomologist’s score. Nonetheless, the data collected across multiple years have not been assessed to consider the overall comparability between the volunteer-derived water quality metrics and the entomologist-derived metrics. Such information has potential to inform usability of volunteer data for management decisions, based on overall comparability of water quality metrics between volunteers and a professional entomologist.

As such, this study was conducted to assess quality of volunteer QC samples and to compare water quality metrics derived from volunteer identification of macroinvertebrates in QC samples of the EQI SMIE project between 2011 and 2016 to the same metrics derived when identification was carried out by a professional entomologist. The following research questions were posed:

* To what extent were macroinvertebrate QC samples prepared and preserved as required for later identification by an entomologist?

* How did commonly used water quality metrics (i.e., SMIE Biotic Index, percent Ephemeroptera, Plecoptera and Trichoptera, and Taxa Richness) compare between QC samples analyzed by volunteers and the same samples identified by a professional entomologist?

* If differences in identification were observed within specific taxa groups, how did that impact overall water quality assessments, and what were the underlying explanations?

Methods

EQI training and sampling protocols

The raw data used in this study were obtained from the EQI’s SMIE biomonitoring efforts. To participate in the project, each volunteer completes 6–8 hours of training that includes both classroom and field-based components. Training is provided under the guidance of the EQI staff, experienced group leaders, and professional freshwater entomologists. Volunteers are guided to use various identification resources during the classroom portion of the training, such as PowerPoint presentations and preserved specimens, which they can observe with dissecting microscopes. This helps volunteers begin to understand three sampling methods to be followed, and familiarizes them with the EQI-developed identification key (Supplemental File 2: EQI SMIE Identification Guide). Volunteers learn how to make naked-eye assessments of the macroinvertebrates’ traits in the field and how to identify them to taxonomic groups designated by the EQI. Volunteers split into groups to practice collecting kick net and leaf pack samples and conducting visual surveys in a stream. Streamside, volunteers identify collected live organisms with the help of the EQI’s training team. After completion of the training, volunteers may sign up for volunteer shifts at the EQI’s monitoring sites.

The EQI volunteers conduct surveys twice a year: once in the spring and once in the fall. At every site, a group leader is paired with other regular volunteers, and the stream is surveyed using three collection methods: kick net, leaf pack, and visual survey. These are kept separate for processing. Group leaders are responsible for overseeing the implementation of the EQI’s sampling and counting protocol at all the EQI monitoring sites, assisting with logistics, and acting as the final authority on field identification of all specimens. The EQI provides additional training for group leaders, which includes evaluations of both macroinvertebrate identification and protocol proficiency. The volunteers must score 93% on an assessment of invertebrate identification, correctly identifying 28 out of 30 live and preserved specimens, to be considered for the role of a group leader. While the program is designed for non-professionals, many SMIE group leaders are local college professors, high school teachers, environmental consultants, retired ecologists, or staff of the EQI and other partner organizations, with some level of prior macroinvertebrate identification experience.

During field monitoring, volunteers collect kick net samples by initially selecting a riffle in the stream in which the water is shallow enough for two volunteers to wade safely. Kick nets with a 500 μm mesh are attached to two poles. The downstream volunteer installs the kick net by placing the two poles in the stream bed perpendicular to the stream bank and at a roughly 45° angle to the stream bed. The downstream volunteer ensures that the bottom of the net is in contact with the bottom of the stream. The bottom of the net is weighted down with several stones. The upstream volunteer uses their feet to kick the stream bed in a rectangle approximately one meter wide across the stream and 1.5 meters long upstream of the net for one minute. The net is moved to shore to a portable camping table or white plastic sheet, and observed aquatic macroinvertebrates are removed from the net by volunteers during a 20-minute sorting period. Organisms are collected from the net with forceps and sorted into ice cube trays filled with stream water.

To conduct leaf pack sampling, one volunteer takes an 18-quart plastic pan and walks up and downstream of the kick net location for three minutes. During that time, the volunteer removes masses of decomposing organic debris caught in the streambed behind rocks or logs and places them in the pan. Stream water is added to the pan and agitated to dislodge organisms from the organic material. The water in the pan containing the organisms is strained through a 500 μm bucket sieve. One volunteer spends 10 minutes searching both the sieve and remaining leaf pack material for organisms, which are placed in a separate ice cube tray from the kick net sample.

The final sample collected is a visual survey to manually search for more cryptic taxa not yet collected in the kick net or leaf pack. To collect this type of sample, the group leader searches for organisms in other ecological niches that are difficult to access through kick net sampling or leaf pack sampling (e.g., submerged rocks, felled logs, and root masses along the embankments) for five minutes. Any visible organisms are removed from the substrate and placed in a white pan filled with stream water for identification.

After all of the organisms are collected, the volunteers and group leader use an EQI-developed identification guide to identify organisms to one of 43 EQI-defined morphological groups (Table 1; Supplemental File 2: EQI SMIE Identification Guide). Identification is completed in the field without the aid of microscopes, though handheld magnifying lenses are sometimes used. These groups were named using descriptive titles easy for non-professionals to understand. They generally represent a family or group of families that are physically similar and occupy similar niches in the stream ecosystem. The volunteers record simple abundance counts per group.

[Table Omitted: See PDF]

For quality assurance purposes, group leaders are required to preserve organisms collected at one site per season (spring and fall) for each of the three sampling methods (i.e., kick net, leaf pack, and visual inspection). Group leaders place organisms in this QC sample into vials containing 70% ethanol. Samples collected using different sampling methods (i.e., kick net, leaf pack, and visual survey) are kept separate. The vials are then returned to the EQI for identification by professional entomologist Dr. Dave Penrose. If any specimens are not preserved (e.g., too rare, large, or predatory), volunteers are instructed to note them on the data sheet (Supplemental File 3: EQI Stream Ecology Sampling Training Guide). In addition, any other issues (such as missing labels in vials, missing one sample type, or an illegible data sheet) are noted by the EQI staff members at time of intake from volunteers.

In the lab, Dr. Penrose uses the same identification guide (Supplemental File 2: EQI SMIE Identification Guide) provided to volunteers to identify all preserved specimens. He examines the samples with a dissecting microscope and enumerates and records abundance of each macroinvertebrate taxonomic group for every QC sample submitted. Any group leader whose identification percentages are below 85% is required to retake one of the EQI’s semi-annual training courses to reprise their role as a group leader.

Quality control sample assessment

A total of 357 QC samples were collected by the EQI SMIE group leaders between 2011 and 2016. We assessed these samples for proper preservation, proper sample labeling, and completeness of notes about specimens that were reported but not preserved (e.g., that were too large for the collection vial). We then assigned each sample a high- or low-quality rating. We did this to address our first research question.

Water quality metric comparisons

To address our second research question, we initially calculated three water quality metrics based on collected macroinvertebrates for the QC samples using taxonomic identification results from volunteers and separately for identification results from the entomologist. These water quality metrics included: Taxa Richness, Ephemeroptera, Plecoptera, and Trichoptera (EPT) Index, and SMIE Biotic Index. We calculated these metrics per sample (i.e., each of three sample methodologies per a designated sample site) as well as per sample site by combining samples collected in kick nets and leaf packs at a designated sample site. Once we calculated the water quality metrics, we assessed agreement between those based on volunteer versus entomologist data using linear regression in R.

We calculated Taxa Richness by summing the number of taxonomic groups present in each sample and separately for each sample site (for kick net and leaf pack samples combined). There are 43 taxonomic groups in SMIE protocols, so possible values for this ranged from 0 to 43. Higher Taxa Richness values are associated with higher biodiversity and presumptively better water quality conditions.

Similarly, for each sample and sample site (i.e., kick net and leaf pack samples combined), we calculated the EPT Index by summing the number of taxonomic groups within the macroinvertebrate orders Ephemeroptera, Plecoptera, and Trichoptera for which one or more individuals were identified. In SMIE protocols, there are 19 taxonomic groups within these three orders, so possible values for this index ranged from 0 to 19. Higher EPT Index values are associated with higher biodiversity and presumably better water quality conditions.

The SMIE Biotic Index was established in 2013 (Supplemental File 1: EQI White Paper). When it was developed, tolerance values were assigned to each taxonomic group based on the EPA’s Rapid Bioassessment Protocol (Barbour et al. 1999). These were tailored to the specific taxa found in the mountains of western North Carolina (Dates and Byrne 1997; Supplemental File 1: EQI White Paper). SMIE tolerance values range from 0 to 10, with higher values indicating organisms more tolerant to pollution. Conversely, lower values indicate more sensitive organisms and presumptively better water quality conditions. These tolerance values are used to calculate a SMIE Biotic Index score for each sample by first multiplying the count of individuals observed within a taxonomic group by the assigned tolerance value (Table 1) for that group. These products are then summed across observed taxonomic groups. Finally, that sum is divided by the total count of individuals across taxonomic groups. The resulting SMIE Biotic Index values are then used to categorize water quality at each site into “Excellent” to “Poor” bioclassifications. The SMIE Biotic Index compares favorably to other indices such as the Izaak Walton League multi-metric index, the Virginia Save our Streams index, and the North Carolina Division of Water Resources index (Supplemental File 1: EQI White Paper).

Identification similarity by taxonomic group

Finally, to address our third research question, we assessed the agreement between volunteer and entomologist collected data based on taxonomic group by comparing specimen counts across all samples using the Bray-Curtis Dissimilarity Index (Bray and Curtis 1957; Kosmala et al. 2016). Dissimilarity in counts of individual specimens within a taxonomic group across all samples for volunteers versus the entomologist was calculated using Equation 1.

[ Equation omitted. See PDF ]

Equation 1: Bray-Curtis Dissimilarity index

In this equation, the S variables are the total number of specimens for a taxonomic group counted across all samples by volunteers (Sv) and the entomologist (Se). Variable C is the lesser of the volunteer versus entomologist specimen counts for each sample, summed across all samples. The Bray-Curtis method produces a dissimilarity score expressed as a decimal. We converted dissimilarity to similarity using Equation 2.

[ Equation omitted. See PDF ]

Equation 2: Percent similarity index derived from Bray-Curtis Index (Equation 1)

For an example calculation of percent similarity for the “spiny crawler mayfly” taxonomic group, consider a scenario where two samples were collected. In Sample #1 the volunteers counted 3 “spiny crawler mayflies” and the entomologist counted 4; in Sample 2, volunteers counted 5 and the entomologist counted 3. Sv is 8; Se is 7; C is 6; Bray-Curtis Dissimilarity is 0.2; and percent similarity is 80%.

We also calculated percent similarity between volunteer and entomologist identifications for each sample. This calculation also used equations 1 and 2, but S variables are the total number of specimens for all taxa groups within a sample by volunteers (Sv) and the entomologist (Se). Variable C is the lesser of the volunteer versus entomologist specimen counts for each taxa group within the sample, summed across taxa groups.

Results

Samples of sufficient quality for analysis

Assessment of QC samples resulted in 79.6% (284/357) of the QC samples being rated as high quality (Table 2). Seventy-three samples (21%) were ranked as low quality. Issues with low-quality samples generally related to poor preservation or sample labeling. Of the 284 high-quality samples, 109 were collected in kick nets, 103 in leaf packs, and 72 through visual surveys.

[Table Omitted: See PDF]

Macroinvertebrate taxa abundances

Volunteers identified a total of 23,688 individual aquatic macroinvertebrates in 41 of the 43 taxa groups across all 284 high quality samples. In comparison, the entomologist identified a total of 24,291 aquatic macroinvertebrates within 38 of the 43 taxa groups (Table 1). Neither volunteers nor the entomologist observed alderflies (Sialidae). Volunteers also did not observe any sand snail case caddis (Helicopsychidae). The entomologist did not observe any red midge (Chironomidae), rounded right face snails (Planorbidae), leech (Hirudinidae), or sand and stick case caddis (Leptoceridae). By count of individuals, the most abundant taxa group was Ephemeroptera followed by Trichoptera, Plecoptera, Diptera, and Coleoptera. Megaloptera, Oligochaeta, Odonata, Gastropoda, and Crustacea each made up 1.5% or less of total individuals for both volunteers and the biologist. While percent representation varied slightly between volunteers and the biologist, the order of representation was the same across all of the taxonomic groups.

Water quality metric comparisons

While the volunteers observed slightly more taxa overall and more numbers of sensitive EPT taxa when the entomologist found Taxa Richness and EPT Index to be lower, and fewer numbers of taxa and sensitive taxa when the entomologist observed Taxa Richness and EPT Index scores to be higher, strong linear relationships were observed between volunteer and entomologist-derived scores for each of the three water quality metrics. This relationship occurred when the data were analyzed by individual collection method (Figure 1a) as well as when the data from both the kick net and leaf pack collection methods were pooled at each collection site (Figure 1b). In all cases, combining the data across these two collection methods at each site affected the regression fit but did not affect the overall statistical significance. For instance, the r² for the SMIE Biotic Index increased when data from the two collection methods were combined at each sample site, instead of calculating the SMIE Biotic Index for each collection method separately (Individual samples r² = 0.68; p < 0.005; Figure 1a. Combined dataset r² = 0.95; Figure 1b). On the other hand, the r² values for Taxa Richness and EPT Taxa Index decreased when data from the kick net and leaf packs were combined (Figure 1).

Figure 1 Comparison of macroinvertebrate water quality metrics calculated with observations from volunteer teams versus the entomologist. Panel a includes all high-quality samples (n = 284) for all collection methods, with collection method depicted according to the legend. Panel b combines organisms observed in samples for leaf pack and kick net methods for each site visit where both samples were high quality (n = 102). Dashed lines in each plot are 1:1 and solid lines are linear regressions for which the r2, p values, and equations are displayed on each plot. Values for Taxa Richness and the EPT taxa index were jittered (producing the clusters of points around integer values) so the many values that fall on the same point can be visualized.

Identification similarity by taxonomic group

Across all samples (n = 284), similarity between volunteer and entomologist identifications ranged from 0 to 100%, with a mean similarity of 83% and median of 90% (Figure 2). By taxa group, percent similarity between volunteer and entomologist identification ranged from 0 to 97% across the 41 observed taxonomic groups (Figure 3). Macroinvertebrate Orders Unionoida and Veneroida (Class Bivalvia) and Odonata had the highest similarity in identification between volunteers and entomologist, while Class Clitellata, Trichoptera, and Class Crustacea had the lowest agreement (Figure 3). The taxonomic groups with the highest agreement were spiny turtle mayfly and dragonfly. Five taxonomic groups had a percent similarity score of 0: red midge, rounded right face snail, sand and stick case caddis, sowbug, and sand snail case caddis.

Figure 2 Bray-Curtis similarity for volunteer versus entomologist organism identification by sample. Across all samples (n = 284) the mean similarity was 83%, median was 90% (black line inside the box), minimum was 0, and maximum similarity was 100%.

Figure 3 Bray-Curtis similarity of volunteer versus entomologist counts of specimens across all samples by taxonomic group (depicted by the open black points and labeled by common name). Taxonomic orders (or other taxonomic level where specified) on the X axis are organized by mean similarity of the groups within that order, where the highest mean similarity is on the left and the lowest mean similarity is on the right (mean order similarity depicted by closed black points, with values connected by the black line). In three instances, pairs of groups have equal similarities and appear as one point on the plot (spiny crawler and filter mayfly; water snipe and water worm; snail caddis, and sand stick caddis).

Discussion

Water quality metrics calculated from the majority of volunteer-generated aquatic macroinvertebrate QC samples submitted to EQI between 2011 and 2016 demonstrated strong linear relationships with those same metrics determined by an entomologist (Figure 1). This suggests that despite differences in identification of individuals between volunteers and a professional, water quality indices generally provided similar results. This was also observed by other researchers who compared volunteer and professional water quality metrics (Fore et al. 2001; Storey and Wright-Stow 2017). These findings suggest that the volunteer-generated data in EQI QC samples have adequate accuracy to complement professional data for general stream quality assessment.

When EQI reports the SMIE data to the public, they determine stream health based on an assessment that includes all three collection methods at a particular site (i.e., kick netting, leaf pack sampling, and visual collection). Our initial research question separated these three methods across all sites to maximize the number of samples to facilitate a more accurate picture of the agreement between volunteer identification and professional identification. We found a strong correlation between volunteers and professionals when the data were assessed with samples separated by collection methods across sites and visits (Figure 1a). However, including visual assessments that had small sample sizes increased the variance, which affected the three water quality index relationships differently. For example, there were several samples where the assessment of stream health from the visual inspection alone produced vastly different SMIE Biotic Index values than those calculated using the other two collection methods (green triangles in Figure 1a). In one example, the volunteer identified four gravel coffin case caddisflies, which have an EQI SMIE tolerance value of 0.8. When this sample was examined by a professional, only a single oligochaete (with a tolerance value of 7) was identified. Thus, the large disparity in SMIE Biotic Index score was driven by small sample sizes collected during the visual inspection. When sites were examined holistically (i.e., by excluding the visual sample and combining kick net and leaf pack samples before calculating the SMIE Biotic Index), the wide differences between SMIE Biotic Index scores were tempered leading to a stronger correlation between the values. In contrast, excluding the visual sample and combining the kick net and leaf pack samples decreased the fit of the regression for Taxa Richness and EPT Taxa Index (Figure 1a, b), artificially deflating the r-squared. While the SMIE Biotic Index is more sensitive than Taxa Richness and EPT Taxa Index to discrepancies between volunteer and entomologist identification when sample size is very small (i.e., when visual assessment samples were assessed separately), eliminating the visual assessment samples and then grouping all organisms across kick net and leaf pack samples alleviated this concern. In fact, the SMIE Biotic Index produced the strongest correlation between volunteer and entomologist observations across the three water quality metrics when visual assessment samples were removed and organisms from leaf pack and kick net samples were pooled. Combining samples collected through robust methods from different habitats such as via kick net and leaf pack sampling can be more effective at including the full suite of macroinvertebrates present in streams. This is because varied macroinvertebrate assemblages exist within differing stream habitats such as snags and riffles (Stepenuck et al. 2008) and benthic and edge habitats (Gill et al. 2024).

The combination of lecture style and hands-on training, and cooperation of experienced volunteer group leaders with the trained volunteers seemed to enable the volunteer teams to effectively preserve and identify these animals. This was represented by an average 83% similarity between volunteer- and entomologist-derived identifications. This aligns with findings from similar participatory science projects. For instance, when volunteers assessed other types of species across ecosystems, they were commonly able to achieve 70% to 95% accuracy as compared to professionals (Swanson et al. 2016).

While the linear relationship was strong between volunteer- and entomologist-derived water quality metrics, volunteers tended to slightly overestimate Taxa Richness and percent EPT taxa when fewer taxa were present, but to underestimate these metrics when greater numbers of taxa groups were present. This may have been due to a desire on the part of the volunteers to find as many organisms as possible to achieve as high a water quality score as possible, especially in low-quality sites, something akin to social desirability bias in a survey (Grimm 2010). This is somewhat similar to findings of a study of lady beetles, where higher species richness was reported by volunteers than professionals (Gardiner et al. 2012). This was attributed to underreporting of common species and overreporting of less common species (Gardiner et al. 2012).

Some general patterns suggest possible explanations for differences in agreement across taxa groups. First, there were no taxa groups that were correctly identified 100% of the time. This suggests that there may be a limit on the level of identification accuracy that volunteers could gain through the training they received. Volunteers more accurately identified invertebrates that tended to be more charismatic or those with recognizable patterning. For example, Unionoidae and Veneroida, clams and mussels, are, by nature of their shells, very identifiable. Dragonflies and damselflies ranked second highest for accurate identification (Figure 3). These animals have distinct shapes that are generally easy to distinguish from other taxa, and are larger than many of the other taxa, making them easier to see without the aid of a microscope or other visual aid. This kind of bias has been observed in other participatory science programs in which volunteers more accurately identified well-known or highly distinct species (Barbato et al. 2021) than lesser-known species or species that appeared very similar to other species (Swanson et al. 2016). For instance, smaller and less distinct taxa such as midges (Chironomidae) and other maggot-like Diptera of various families, mites, micro caddisflies (Hydroptilidae), oligochaetes, and fingernail clams (Spharidae) were observed to be recorded less often by volunteers than professionals in one study (Storey and Wright-Stow 2017). To address this finding, participatory science programs like EQI should consider offering follow-up trainings that focus on smaller or less charismatic specimens to boost the ability of volunteers to correctly identify those taxa.

Another difference between volunteer and professional taxonomic identification worth noting is that no red midges were identified by the professional, whereas some were identified by the volunteers. This is likely due to preservation time and the influence of ethanol on color of the specimens. Green chironomids are more likely to keep their color when preserved, whereas red midges would lose their color over time (Epler 1987). This is a form of sampling bias that may need to be addressed in the EQI’s sampling training or protocol. It is also possible that there were errors made when recording data during field identification. Typically, the group leaders call out an identification and a count of individual organisms and another volunteer tallies the final total. This allows for the possibility of transcription errors and mathematical errors, although our data indicate that, overall, these errors have little impact on the accuracy of stream quality assessment (Figure 1).

Still another notable difference between volunteers and the entomologist was in the lack of similarity between sowbug identifications. In fact, there was 0% similarity between volunteers and the entomologist. While volunteers identified eight sowbugs across the QC samples and the entomologist identified two, further inspection of the data revealed that the sowbugs identified by the entomologist were not in the same samples where volunteers identified sowbugs. In one instance, the issue was caused by datasheet legibility, and the entomologist actually did identify the sowbugs, but the record was not captured in the QC dataset. We are uncertain about the other instances, but believe the explanations center on the very low frequency that these organisms are observed in the study area. For samples where the entomologist observed a sowbug and the volunteers did not, it is possible that the sowbugs had “hitchhiked” into a preserved sample by clinging to another specimen. As the volunteers generally identified macroinvertebrates without the aid of a magnifying lens or microscope, having one or two specimens preserved but not observed by the volunteers is not surprising. Conversely, for specimens identified by volunteers but not observed by the entomologist, we speculate that volunteers may not have included those specimens in the preserved samples. Their QC methods prescribe them to return rare specimens to the stream rather than to preserve them in the sample. Since so few sowbugs were observed across the 357 samples, we speculate these organisms may have been returned to the stream by the volunteers prior to sample preservation. As such, the organisms were not present in the sample for the entomologist to identify.

The difference in numbers of individual macroinvertebrates identified between volunteers and the entomologist is also important to consider. In about 75% of samples, volunteers identified more specimens than the entomologist. There are a variety of possible reasons such differences may have resulted. First, improper, or poor preservation of macroinvertebrates may have resulted in fewer specimens available for the entomologist to identify. The 70% ethanol solution used for preservation was significantly less concentrated than the 95% solution recommended by the US Geological Survey to protect specimens from decay (Keith et al. 2022). In addition, if even small amounts of organic matter saturated with stream water were inadvertently included in a sample, that would reduce preservation strength further, potentially rendering the ethanol solution unsuitable for long-term storage of many of the specimens. There is no location on the entomologist’s data sheet to record specimens unable to be identified, thus such data would be lost in the entomologist’s comparison data. Further support for this idea is that many (17% of the 21%) of samples identified as having low quality were due to preservation issues. Conversely, where the entomologist identified more specimens than the volunteers (in about 25% of samples), this may have been due to small specimens “hitchhiking” on other specimens. As volunteers identify macroinvertebrates live, the macroinvertebrates may attempt to take cover by clinging to whatever materials are present, which may include other macroinvertebrates. Also, as volunteers identified primarily only with the naked eye, they may have missed observing specimens attached to other specimens. Recommendations to improve comparability between number of individual macroinvertebrates identified by volunteers and the entomologist in the future include using a 95% ethanol solution to preserve specimens and encouraging volunteers to use the hand lens they are provided to increase their ability to observe hitchhikers. In addition, as related to the low-quality QC samples, EQI and similar volunteer monitoring programs might consider sending reminders to group leaders to preserve (and consider re-preserving with fresh ethanol) QC samples close in time to when QC samples are collected.

It is also worth noting that many of the samples with the weakest agreement between volunteers and professionals came from the visual inspection methodology (Figure 1a, green triangles). While this makes some sense as the number of organisms able to be collected by hand by a single person in five minutes will be lower than bulk collection methods such as kick netting, inclusion of these samples can have significant effects on volunteer and professional biologist agreement. This is especially true in situations where only one or two organisms are collected. This increases the chance for significant disagreements between water quality metrics reported by volunteers and professionals. As a result, and supported through our data analyses, we recommend excluding visual samples from future sampling and water quality metric calculations.

Conclusions

This study demonstrates that trained volunteers can identify macroinvertebrates adequately to generate water quality scores with strong relationships to those generated by a professional entomologist. This is particularly true when macroinvertebrate subsamples from kick nets and leaf packs were combined and when data collected through visual assessments were excluded during calculation of water quality metrics. While volunteer macroinvertebrate identification had an overall similarity of 83% to professional identification, the study highlighted the importance of volunteer training and the value of having experienced group leaders on board to support volunteer macroinvertebrate collections to ensure effective preservation and identification of macroinvertebrates. Having a QC program in place is important to detect aspects of methods where further training is warranted. These results are important to the field of participatory science as they inform volunteer macroinvertebrate monitoring methods and training needs that, if implemented, can help improve the comparability of volunteer and professional water quality assessments. Further, our results provide evidence of the validity of volunteer-generated data, which enhances their ability to be used to extend limited agency resources to monitor water quality.

Data Accessibility Statement

Data and code available as a resource on Hydroshare (Sigler, Hamilton, and Traylor 2024).

Supplementary Files

The supplementary files for this article can be found as follows:

Supplemental File 1

EQI White Paper. DOI: https://doi.org/10.5334/cstp.756.s1

Supplemental File 2

EQI SMIE Identification Guide DOI: https://doi.org/10.5334/cstp.756.s2

Supplemental File 3

EQI Stream Ecology Sampling Training Guide DOI: https://doi.org/10.5334/cstp.756.s3

Acknowledgements

We thank the volunteers who participated in macroinvertebrate quality control sampling for EQI from 2011 through 2016 and the dedicated volunteers for the program who have donated their time and knowledge to the program for so many years. We also thank Noelle Hasan for assistance in organizing references.

Funding Information

The Pigeon River Fund of the Community Foundation of Western North Carolina has provided the majority of annual funding for the SMIE program.

Competing Interests

The authors have no competing interests to declare.

Author Contributions

Hamilton: Conceptualization, Methodology, Writing – Original Draft, Data Curation, Editing. Stepenuck: Review and Editing. Zinna: Conceptualization, Data Curation, Review and Editing, Resources. Traylor: Resources, Review and Editing. Penrose: Review and Editing, Validation. Sigler: Conceptualization, Methodology, Formal Analysis, Visualization, Supervision, Review and Editing.

References

1. Barbato, D., Benocci, A., Guasconi, M., and Manganelli, G. (2021) Light and shade of citizen science for less charismatic invertebrate groups: Quality assessment of iNaturalist nonmarine mollusc observations in central Italy. Journal of Molluscan Studies, 87: eyab033. DOI: https://doi.org/10.1093/mollus/eyab033

2. Barbour, M.T., Gerritsen, J., Snyder, B.D., and Stribling, J.B. (1999) Rapid bioassessment protocols for use in streams and wadeable rivers: Periphyton, benthic macroinvertebrates and fish, second edition. EPA 841-B-99-002. Washington D.C. https://www3.epa.gov/region1/npdes/merrimackstation/pdfs/ar/AR-1164.pdf

3. Bray, J.R. and Curtis, J.T. (1957) An ordination of the upland forest communities of southern Wisconsin. Ecological Monographs, 27, pp. 326–349. DOI: https://doi.org/10.2307/1942268

4. Cao, Y., and Hawkins, C.P. (2011) The comparability of bioassessments: A review of conceptual and methodological issues. Journal of the North American Benthological Society, 30, pp. 680–701. DOI: https://doi.org/10.1899/10-067.1

5. Cook, S.E.K. (1976) Quest for an index of community structure sensitive to water pollution. Environmental Pollution (1970), 11, pp. 269–288. DOI: https://doi.org/10.1016/0013-9327(76)90067-7

6. Dates, G. and Byrne, J. (1997) Living waters: Using benthic macroinvertebrates and habitat to assess your river’s health. River Watch Network.

7. Engel, S., and Voshell, J.R. (2002) Volunteer biological monitoring: Can it accurately assess the ecological condition of streams? American Entomologist, 48, pp. 164–177. DOI: https://doi.org/10.1093/ae/48.3.164

8. Epler, J.H. (1987) Notes on the Dicrotendipes (Diptera: Chironomidae) of Mexico, with descriptions of two new species. Entomologica Scandinavica Supplement, 29:147–154.

9. Feeley, H.B., Woods, M., Baars, J.R., and Kelly-Quinn, M. (2012) Refining a kick sampling strategy for the bioassessment of benthic macroinvertebrates in headwater streams. Hydrobiologia, 683, pp. 53–68. DOI: https://doi.org/10.1007/s10750-011-0940-9

10. Fore, S.L., Paulsen, K., and O’Laughlin, K. (2001) Assessing the performance of volunteers in monitoring streams. Freshwater Biology, 46, pp. 109–123. DOI: https://doi.org/10.1111/j.1365-2427.2001.00640.x

11. Gardiner, M.M., Allee, L.L., Brown, P.M.J., Losey, J.E., Roy, H.E., and Smyth, R.R. (2012) Lessons from lady beetles: Accuracy of monitoring data from US and UK citizen science programs. Frontiers in Ecology and the Environment, 10, pp. 471–476. DOI: https://doi.org/10.1890/110185

12. Gill, B.A., Allen, D.C., Mims, M.C., Neeson, T.M., Ruhi, A., Atkinson, C.L., Shogren, A.J., et al. (2024) Combined benthic and stream edge sampling better represent macroinvertebrate assemblages than benthic sampling alone along an aridity gradient. Limnology and Oceanography: Methods. 22, pp. 208–216. DOI: https://doi.org/10.1002/lom3.10601

13. Gowan, C., Ruby, M., Knisley, R., and Grimmer L. (2007) Stream monitoring methods suitable for citizen volunteers working in the Coastal Plain and Lower Piedmont regions of Virginia. American Entomologist, 53, pp. 48–57. DOI: https://doi.org/10.1093/ae/53.1.48

14. Grimm, P. (2010) Social Desirability Bias. Wiley International Encyclopedia of Marketing. DOI: https://doi.org/10.1002/9781444316568.wiem02057

15. Guareschi, S., Laini, A., and Sánchez-Montoya, M.M. (2017) How do low-abundance taxa affect river biomonitoring? Exploring the response of different macroinvertebrate-based indices. Journal of Limnology, 76, pp. 9–20. DOI: https://doi.org/10.4081/jlimnol.2016.1516

16. Haase, P., Murray-Bligh J., Lohse, S., Pauls, S., Sundermann, A., Gunn, R., and Clarke, R. (2006) Assessing the impact of errors in sorting and identifying macroinvertebrate samples. Hydrobiologia, 566, pp. 505–521. DOI: https://doi.org/10.1007/s10750-006-0075-6

17. Keith, B.R., Carleen, J.D., Larson, D.M., Anteau, M.J., and Fitzpatrick M.J. (2022) Protocols for collecting and processing macroinvertebrates from the benthos and water column in depressional wetlands (No. 2022-1029). US Geological Survey. (Available from: https://pubs.usgs.gov/publication/ofr20221029/full)

18. Kosmala, M., Wiggins, A., Swanson, A., and Simmons, B. (2016) Assessing data quality in citizen science. Frontiers in Ecology and the Environment, 14, pp. 551–560. DOI: https://doi.org/10.1002/fee.1436

19. Krabbenhoft, C.A., and Kashian, D.R. (2020) Citizen science data are a reliable complement to quantitative ecological assessments in urban rivers. Ecological Indicators, 116, 106476 [106476 is an article number not a page number so I deleted “p.”]. DOI: https://doi.org/10.1016/j.ecolind.2020.106476

20. Li, L., Zheng B., and Liu L. (2010) Biomonitoring and bioindicators used for river ecosystems: Definitions, approaches and trends. Procedia Environmental Sciences, 2, pp. 1510–1524. DOI: https://doi.org/10.1016/j.proenv.2010.10.164

21. Maas, R.P., Kucken, D.J., and Gregutt, P.F. (1991) Developing a rigorous water quality database through a volunteer monitoring network. Lake and Reservoir Management, 7, pp. 123–126. DOI: https://doi.org/10.1080/07438149109354262

22. Moffett, E.R., and Neale, M.W. (2015) Volunteer and professional macroinvertebrate monitoring provide concordant assessments of stream health. New Zealand Journal of Marine and Freshwater Research, 49, pp. 366–375. DOI: https://doi.org/10.1080/00288330.2015.1018913

23. NC DEQ. (2016) Standard Operating Procedures for the Collection and Analysis of Benthic Macroinvertebrates. Available at: https://files.nc.gov/ncdeq/Water%20Quality/Environmental%20Sciences/BAU/NCDWRMacroinvertebrate-SOP-February%202016_final.pdf (Accessed 1 November 2024).

24. NC DEQ. (n.d.) Water Quality Data Tiers. Available at: https://www.deq.nc.gov/water-quality/planning/tmdl/303d/general/data-tiers/download (Accessed 1 November 2024).

25. Nerbonne, J.F., and Vondracek B. (2003) Volunteer macroinvertebrate monitoring: Assessing training needs through examining error and bias in untrained volunteers. Journal of the North American Benthological Society, 22, pp. 152–163. DOI: https://doi.org/10.2307/1467984

26. Nerbonne, J.F., Ward, B., Ollila, A., Williams, M., and Vondracek, B. (2008) Effect of sampling protocol and volunteer bias when sampling for macroinvertebrates. Journal of the North American Benthological Society, 27, pp. 640–646. DOI: https://doi.org/10.1899/07-101.1

27. Oertel, N., and Salánki J. (2003) Biomonitoring and bioindicators in aquatic ecosystems. In: R.S. Ambasht and N. K. Ambasht (Eds). Modern Trends in Applied Aquatic Ecology. New York: Springer. pp. 219–246. DOI: https://doi.org/10.1007/978-1-4615-0221-0_10

28. O’Leary, N., Vawter, A.T., Wagenet, L.P., and Pfeffer, M. (2004) Assessing water quality using two taxonomic levels of benthic macroinvertebrate analysis: Implications for volunteer monitors. Journal of Freshwater Ecology, 19, pp. 581–586. DOI: https://doi.org/10.1080/02705060.2004.9664738

29. Peeters, E.T.H.M., Gerritsen, A.A.M., Seelen, L.M.S., Begheyn, M., Rienks, F., and Teurlincx, S. (2022) Monitoring biological water quality by volunteers complements professional assessments. PLoS ONE, 17:e0263899. DOI: https://doi.org/10.1371/journal.pone.0263899

30. Pinto, P., Oliveira-Junior, J.M.B., Leitão, F., Morais, M.M., Chícharo, L., Vaz, P., … and Teodósio, M.A. (2020) Development of a metric of aquatic invertebrates for volunteers (MAIV): A simple and friendly biotic metric to assess ecological quality of streams. Water, 12(3), 654. DOI: https://doi.org/10.3390/w12030654

31. Poff, N.L., Olden J.D, Vieira N.K.M., Finn D.S., Simmons M.P., and Kondratieff B.C. (2006) Functional trait niches of North American lotic insects: traits-based ecological applications in light of phylogenetic relationships. Journal of the North American Benthological Society, 25, pp. 730–755. DOI: https://doi.org/10.1899/0887-3593(2006)025[0730:FTNONA]2.0.CO;2

32. Postelle, B. (2009) Director seeks new home for water-testing center cut by UNCA. Mountain Xpress, 26 August. Available at: http://mountainx.com/news/082609director_seeks_new_home/ (Accessed 1 November 2024).

33. Ross, K. (2020) Bracing for budget cuts: Slashing expected at NC Department of Environmental Quality. Raleigh, North Carolina: Carolina Public Press, 3 August. Available at: https://carolinapublicpress.org/36865/nc-budget-stalemate-cuts-department-environmental-quality-deq/ (Accessed 1 November 2024).

34. Sigler, W.A., Hamilton, V., Traylor, A.M. (2024). Volunteer accuracy in a benthic macroinvertebrate participatory science project – data and code, Hydroshare. Available at: 10.4211/hs.3fa46a0a96cb47219aab2230ad141a42 (Accessed 4 November 2024).

35. Smith, J.G., Brandt, C.C., and Christensen S.W. (2011) Long-term benthic macroinvertebrate community monitoring to assess pollution abatement effectiveness. Environmental Management, 47, pp. 1077–1095. DOI: https://doi.org/10.1007/s00267-010-9610-3

36. Storey, R.G., and Wright-Stow, A. (2017) Community-based monitoring of New Zealand stream macroinvertebrates: agreement between volunteer and professional assessments and performance of volunteer indices. New Zealand Journal of Marine and Freshwater Research, 51, pp. 60–77. DOI: https://doi.org/10.1080/00288330.2016.1266674

37. Stepenuck, K.F., Crunkilton, R.L., Bozek, M.A., and Wang, L. (2008) Comparison of macroinvertebrate-derived stream quality metrics between snag and riffle habitats. Journal of the American Water Resources Association, 44, pp. 670–678. DOI: https://doi.org/10.1111/j.1752-1688.2008.00197.x

38. Stribling, J.B., Pavlik, K.L., Holdsworth, S.M., and Leppo, E.W. (2008) Data quality, performance, and uncertainty in taxonomic identification for biological assessments. Journal of the North American Benthological Society, 27, pp. 906–919. DOI: https://doi.org/10.1899/07-175.1

39. Swanson, A., Kosmala, A.M., Lintott, C., and Packer, C. (2016) A generalized approach for producing, quantifying, and validating citizen science data from wildlife images. Conservation Biology 30, pp. 520–531. DOI: https://doi.org/10.1111/cobi.12695

40. Wallace, J.B. and Webster, J.R. (1996) The role of macroinvertebrates in stream ecosystem function. Annual Review of Entomology, 41, pp. 115–139. DOI: https://doi.org/10.1146/annurev.en.41.010196.000555