Do observer fatigue and taxon bias compromise visual encounter surveys for small vertebrates?

Additional keywords: applied ecology, conservation, invasive species, wildlife management.

Visual encounter surveys are an important and widely used animal sampling tool, providing the data necessary to monitor populations and quantify the impacts of environmental change (Foster et al. 2012). Unfortunately, detections and counts may depend on a wide variety of factors besides size and/or density of the focal population. Biases may be linked to the organism itself (e.g. sex effects, size effects or other sources of individual heterogeneity in activity and microhabitat use; Tyrrell et al. 2009; Christy et al. 2010), to spatially and temporally varying environmental conditions (abiotic and/or biotic, such as vegetation structure; Smith et al. 1995; Anderson et al. 2015; Lardner et al. 2015) or to the surveyors’ skills (Rodda and Fritts 1992; Thompson and Mapstone 1997; Diefenbach et al. 2003). The latter can be a combination of acquired search image, vision acuity and ability to focus on the task for an extended period of time.

Some studies have deployed decoys in known numbers and locations to accurately assess spatial detection patterns and observer biases (Smith et al. 1995; Lardner et al. 2007; Kanary et al. 2010). However, surveying for decoys may not be directly comparable to surveying for real animals, due to microhabitat differences, movement cues from real animals and differing light reflectance of real animals versus decoys. If the surveyed animals are highly visible, immobile (e.g. sleeping) and undisturbed by visual observers, they can provide an ideal and unique situation for assessing observer-related biases.

Visual surveys constituted a major part of a 2016 multi-agency rapid response (cf. Stanford and Rodda 2007) effort to the west Pacific island of Saipan, employed to detect a possible incursion by the invasive, nocturnal brown treesnake (Boiga irregularis). This snake is abundant on Guam, ~185 km to the south-south-west of Saipan, but is not known to be established on Saipan or any other island of the Commonwealth of the Northern Mariana Islands (CNMI). Many of the participants searched on foot for snakes and their potential prey 5–6 nights per week, often for several weeks on end. Performing monotonous tasks for an extended time period can lead to fatigue and performance failures (Stern and Bynum 1970; Ransom 2012). As time-on-task increases, fatigue may influence the ability and decision to initiate eye movement, and thereby diminish the positive detection effect of practice on a task.

During nocturnal surveys on Saipan we recorded >2.3 × 10⁴ reptile and small mammal sightings, mostly diurnal lizards seen sleeping on their arboreal night perches. We used this dataset to assess if observer skills suffered from time-on-task during the 4 h of nightly surveys, because the sleeping lizards did not move over the course of an evening and the observers’ circadian rhythm was therefore not confounded with the targets’ circadian rhythm (cf. Lardner et al. 2015; Rodda et al. 2015).

Observer fatigue (or boredom) applies to nightly survey duration and time of night, but may also occur when someone has been surveying for prolonged periods (i.e. weeks). Whereas some observers only participated for a few nights, others searched for up to 31 nights (spread out over two periods separated by a month-long break). This allowed us to assess if sighting rates were affected by how many nights a person had previously surveyed.

Because of the large number of observations (14 out of 29 observers each recorded >1000 reptile and mammal sightings), we also assessed biases in what taxa each observer detected to a higher-versus-lower degree, compared with the pool of peer observers.

Surveys took place in the vicinity of the Saipan airport and Dandan village (~15.127°N, 145.735°E) from 6 January to 13 March 2016, 6–7 nights per week, but with a break in surveying from 26 January through 21 February. For analyses we used data on selected vertebrate taxa (lizards and small mammals, both potential snake prey) recorded from survey transects with similar vegetation structure, i.e. secondary forest dominated by the introduced leguminous tree, Leucaena leucocephala; emergent trees were partially denuded from typhoon Soudelor, which had passed over Saipan on 2 August 2015. Bird counts suffered from biases of no interest to us (sometimes a bird was flushed by the first person passing it during the evening; two species of terns, which made up 60% of all bird records, were often detected due to their vocal congregations in tall trees), and were therefore omitted from analysis.

During 38 evenings (35 of which are analysed here) a varying number of persons wearing powerful LED (light-emitting diode) headlamps searched for snakes perched in the vegetation or on the ground along roads and paths. Each observer only searched on one side of a road or trail and was instructed to catch any snake encountered (none were seen) and record all vertebrates detected. Transects had been measured and flagged with regular distance markers, and observers took notes on the distance they walked along each transect. They also recorded start and end times, and total actual search time. The mean ± s.d. length of the 17 transects we analyse here was 0.42 ± 0.04 km (range 0.34–0.51 km) and observers were instructed to spend ~1 h on each transect. Usually four transects were assigned to each observer on a given night, and searches took place between ~1900 and 2300 hours. A transect was usually searched once or twice on a given night, rarely more. Repeated surveys on the same night were always done by different people and were usually separated by a 1-h time slot.

The primary foci for our analyses were two introduced, diurnal and arboreal lizards that were frequently observed sleeping in elevated positions: the emerald tree skink (Lamprolepis smaragdina) and the green anole (Anolis carolinensis). Our surveys started at dusk (see Fig. S1, available as supplementary material to this paper), and we have no indication that these lizards changed position during an evening; they remained motionless even when closely approached with a strong light shining on them.

Four species of geckos were recorded: Asian house gecko (Hemidactylus frenatus), mourning gecko (Lepidodactylus lugubris), oceanic gecko (Gehyra oceanica) and mutilating gecko (Gehyra mutilata). Identification skill differences among observers and the (at times) long distance between observer and gecko frequently resulted in the inability to identify to species; these individuals were recorded generically as ‘gecko’. These taxa are of reasonably similar size and colour, and all are primarily arboreal and nocturnal; thus we lumped all geckos together and analysed sightings as a group.

Sightings of nocturnally active small mammals were dominated by rats (Rattus sp.), followed by musk shrews (Suncus murinus), house mice (Mus musculus) and unidentified small mammals. We lumped them all into the group ‘small mammals’. Because both groups are nocturnal, changes in sighting rates of geckos and small mammals over the course of an evening can be attributed to a combination of observer fatigue and the circadian rhythm of the focal taxon/taxa.

We analysed variation in the number of target organisms detected along each transect walked by one observer. Which transect the data originated from was of no inherent interest to us, and merely a nuisance variable. Because our statistical models controlled for differences among transects, any consistent differences in their respective length, as well as variations in walking pace stemming from the desired 1-h-per-transect duration, were of no concern. However, surveys were occasionally cut short, and there were (rare) cases of longer-than-intended distances walked. There were also occasions when an observer walked unusually quickly or slowly. As a compromise between excessive data pruning and undesired noise we opted to omit transect surveys with an actual distance less than 85% of, or more than 117.65% [(1/0.85) × 100] of, the median (= modal) recorded distance for the focal transect. We also excluded transects walked at a pace >150% or <66.67% of the mean (0.45 km h^–1). Additional data were omitted because of missing distance records or notes on electronic data entry issues. This left us with 920 transect searches, contributed unevenly by 29 observers who collectively walked 386.8 km (Table 1).

Table 1.  Visual survey efforts by 29 observers and their animal detections
Observers are listed in declining order of effort contributed to the survey. Columns indicating taxon-specific biases draw from binomial proportion tests (see text) and indicate the Bonferroni-corrected significance level of the focal observer’s detection bias for (↑) or against (↓) the focal taxon. The numbers of individuals detected per kilometre are simple means. ***P < (0.001/116) = 0.00000862; **P < (0.01/116) = 0.0000862; *P < (0.05/116) = 0.000431; n.s., not significant (P > (0.05/116) = 0.000431)

To evaluate if observer fatigue may have caused a decline in the number of animals detected as the evening progressed, we estimated in R (https://www.r-project.org, accessed 22 September 2016; package lme4) mixed Poisson models containing a fixed intercept and a fixed time-of-night effect, random intercepts for factors transect and night, and for factor observer we specified both a random intercept and a random time-of-night associated slope (thus allowing both ‘skill’ and nightly fatigue effects to vary among observers). Fixed-effect coefficients, as well as random effects associated with the observers, were extracted and processed in a spreadsheet to illustrate the overarching (fixed) time-of-night effect as well as (random) across- and within-observer patterns. We did separate analyses for four response variables: counts of (1) the diurnal skink L. smaragdina; (2) the diurnal iguanid A. carolinensis; (3) predominately nocturnal gecko taxa pooled; and (4) predominately nocturnal small mammals (rats, mice and shews) pooled.

To make the random observer intercepts reflect observer skills despite the random slopes, we transformed (before model estimation) the time-of-night data so that each transect time stamp (the midpoint between the transect’s start and end time) was expressed as a deviation, in decimal hours, from the global mean (2056 h). Any within-observer fatigue effect is then pivoted around that observer-specific mid-evening ‘skill’ level, and observer skill ranks are not confounded by their (respective) fatigue-effect slopes.

We compared the statistical significance of the time-of-night effects in the mixed models with similar effects derived from fixed-effects-only models. These models included as fixed main effects variables time-of-night , transect, night, observer and an intercept. We did not include any observer–by–time-of-night effect as that would have restricted the main time-of-night effect to the single observer that was coded as the reference state and a staggering number (n = 28) of interaction effects.

In a second set of analyses we adjusted the time-of-night data because surveys commenced at a progressively earlier point after sunset as the northern hemisphere winter turned into spring (January to March; Fig. S1). These analyses tried to address if any time-of-night effects might be better explained by the target organisms’ sunset-related circadian rhythm than our surveyors’ season-invariant search schedule (and presumed season-independent search fatigue). After this ‘relative-to-sunset’ adjustment, transect midpoint data no longer had a global mean of zero.

To evaluate the pool of observers’ detection rates over longer time periods, we coded variable deploymentnight as ordinal integers specific to each participant. The value for an observer’s focal night took into account any nights that she/he searched transects other than those analysed here (causing gaps in the surveyor’s deploymentnight number series), while ignoring any breaks (days or weeks) in that person’s effort. Because some observers searched on many nights and others on few, data are richer for low search–night numbers than for high numbers. We analyse the fixed effect of deploymentnight in four taxon-specific models containing the random factors of transect, night and observer. Because the observer pool shifted over time, and because of the frequency with which different observers contributed varied, deploymentnight is only partially confounded by any temporal patterns related to the animals themselves.

Our mixed Poisson models were focused on one taxon (or a taxonomic group) at a time, and did not allow us to test if observers differed in propensity to see some taxa rather than others (i.e. if they were taxonomically biased). To address that question we reverted to the raw count data in Table 1 and conducted binomial proportion tests for each observer and focal taxon (or taxon group). We compared each observer against the pool of other observers (n = 28), asking if the proportion of species ‘x’ out of the focal observer’s total animal sightings differed from the proportion of species ‘x’ among all other pooled observers’ sightings. For example, observer 1 saw 3159 animals of which 1739 (55%) were emerald tree skinks; this was contrasted against the 52% (10 637/20 468) recorded by the other 28 observers. We performed Bonferroni corrections to account for the fact that 29 × 4 = 116 tests were conducted.

Detections of emerald tree skinks exhibited a small but statistically verified decline as the evening progressed (Fig. 1a, Table 2), whereas detections of green anoles increased during the evening (Fig. 1b, Table 2). Detections of geckos showed a weak and non-significant declining trend (Fig. 1c, Table 2), but small mammal sightings (rats, shrews and mice) declined more sharply (by ~25%) over the course of an evening (Fig. 1d, Table 2).

Fig. 1.  Model-predicted number of sightings per transect of (a) sleeping Lamprolepis smaragdina, (b) sleeping Anolis carolinensis, (c) nocturnal geckos and (d) nocturnal small mammals as a function of time-of-night and for 29 observers. The thick dotted line is the fixed-effect prediction; lines for individual observers – whose effort contributions differed – add the effects of their respective random intercepts and random slopes. Random intercepts for transects (n = 17) and nights (n = 35) are held constant at their means (β_i = 0).

Table 2.  Comparison of time-of-night effects in mixed versus fixed-effects-only models
For both model classes, the time-of-night effect was coded as a fixed effect. The mixed-model coefficients represent the time-of-night slopes illustrated by dotted lines in Fig. 1. n.s., not significant

time-of-night effect estimates from models where we used deviations from the mean ‘clock time’ (UTM+10) differed only marginally from models in which we further adjusted for the seasonally changing time of sunset (Table 3). For small mammals, the model with a sunset-adjusted time specification had convergence issues, showing a perfect correlation between the searchers’ random intercepts and time-of-night effects. For the taxa in which Akaike’s Information Criterion (AIC) leaned most heavily towards one time coding rather than the other (green anoles and small mammals, which were the taxa with the strongest time-of-night effects), ‘clock time’ was supported over sunset-adjusted time.

Table 3.  Comparisons of alternative time effects on predicted animal counts
For each taxon (Ls = Lamprolepis smaragdina; Ac = Anolis carolinensis; ge = gecko taxa pooled; sm = small mammal taxa pooled), predictions are shown from mixed models with different (fixed-effect) time specifications: those utilising the unadjusted Chamorro Standard Time (UTC +10) versus a variable adjusting for the progressively later sunset as winter turned into spring. For each time specification and taxon, the predicted count difference (Δ) contrasts the first and last transect (each with a duration of 1 h) of a 4-h search evening, centred around 1930 hours versus 2230 hours. n.s., not significant

Whereas some observers participated in the study for up to 31 nights (spread out over two discrete survey bouts), we observed no decline in sighting rates over the date ranges in which they participated, and therefore no evidence of a long-term ‘deployment fatigue’ effect. On the contrary, the pattern was that more (not fewer) emerald tree skinks and green anoles were sighted the longer a person had been deployed (P < 0.001; Fig. 2a, b). For geckos and small mammals (Fig. 2c, d), there were no such temporal effects (P = 0.83 and P = 0.09, respectively).

Fig. 2.  Impact on model-predicted animal counts per transect of variable deploymentnight, i.e. which ordinal search night for a focal observer from which data originated, ignoring any breaks a person had in her/his surveying but taking into account when surveys were done on transects not included in this analysis. Dotted regression lines are from models treating deploymentnight as a continuous variable. Models treating each deploymentnight as a categorical state render the first night of deployment (ordinal value = 1) the reference state; model-predicted animal counts on nights after that are shown with 95% confidence interval. Panels a–d show the four taxa (groups) in the order (a) Lamprolepis smaragdina, (b) Anolis carolinensis, (c) geckos (all taxa pooled) and (d) small mammals (all taxa pooled); panel e shows the transect-survey n-value for each value of deploymentnight. All models also included random intercepts for categorical observer, transect and night effects; these were held constant at their means (0) when calculating the model-predicted counts.

A majority of observers were biased (compared with the pool of peer observers) in what taxa they detected (Table 1, Fig. 3). This was most evident for those who contributed many sightings and for which there was sufficient statistical power to verify a bias (after heavily penalising Bonferroni corrections). Out of 14 observers that collectively detected >1000 animals each, only two (observers #2 and #15; Table 1) were unbiased in relation to the taxonomic ratio patterns among the 28 peer observers, whereas for observers that detected <1000 animals, no significant bias was evident for five of 15 people. However, no observer was biased for all four taxa (taxa groups). Some appeared to be particularly biased in their emerald tree skink : green anole ratio (i.e. the two taxa of green, diurnal lizards). For example, there were observers who each recorded >1150 green lizards and their skink-to-anole ratios spanned from 0.86 (534 : 619) to 9.50 (1150 : 121)

Fig. 3.  Taxonomic detection bias among the 29 observers. In each panel focusing on a specific taxon or a taxonomic group (Ls = Lamprolepis smaragdina, Ac = Anolis carolinensis, ge = gecko taxa pooled, sm = small mammal taxa pooled), circles represent the proportions of animals recoded by each observer belonging to the focal taxon (group). Notice the different y-axis scales across taxa. The grey scales of circles reflect the statistical significance (or lack thereof) in binomial proportion tests where the focal person’s ratio was contrasted to the ratio of the other 28 persons’ detections pooled (the latter ratios are indicated by dashes to the left in each sub-panel). Significance levels were Bonferroni-corrected because we conducted 29 × 4 = 116 tests and are coded as follows: black = P < (0.001/116) = 0.00000862 ***; dark grey = P < (0.01/116) = 0.0000862 **; light grey = P < (0.05/116) = 0.000431 *; no fill = P > (0.05/116) = 0.000431 n.s.

Although observers were biased in taxa detected, detection rates of some observers appeared to be consistently above average and, conversely, consistently below average in others (Fig. 4), as indicated by the highly significant regression (P < < 0.001; Fig. 4b).

Fig. 4.  Random observer intercepts from four mixed models that focus on different taxa for the 29 observers. Values above the x-axis indicate above-average skills in detecting the focal taxon (taxon group). In panel a, observers are ordered as in Table 1 (i.e. by survey effort contribution) and in panel b by increasing surveyor across-taxa ‘skill’ level (mean of the person’s four random intercepts), with a linear regression fitted to the data.

Disentangling the effects of a target species’ circadian rhythm from that of observer fatigue as a survey progresses into the night can be challenging (Lardner et al. 2015), although among-species comparisons can offer clues to the relative magnitude of these sources of count variation (Rodda et al. 2015). During our search for brown treesnakes in Saipan, the abundant sightings (n = 12 376) of large and shiny emerald tree skinks sleeping in exposed perches offered an unprecedented opportunity to remove the target’s circadian rhythm from the equation, and to analyse how observer fatigue affected lizard counts over the course of our 4-h evening search period. We found a small but statistically verified decline in emerald tree skink counts, suggesting observers became slightly less attentive as the evening progressed. However, the decline was sufficiently small that we do not see any reason to consider shortening search duration for future rapid response efforts.

Surprisingly, we detected significantly more – not fewer – sleeping green anoles as the evening progressed. We have three non-exclusive hypotheses for this pattern. First, although diurnal, perhaps some anoles did not appear in their sleeping positions until well into the evening, and thus were not as readily available for detection when surveys commenced during nautical or astronomical twilight. The poorer model fit for the sunset-adjusted time-of-night coding (Table 3) lends no support to this hypothesis but movement away from night perches before dawn has been reported in Anolis etheridgei (Mahler 2010). Second, whereas green anoles can assume different colours (green or brown) during the day depending on temperature, social circumstances and stress (Carlton 1903), they are nearly always green at night, when asleep (Gordon and Fox 1960). In our experience, the bright green colour of a sleeping anole makes it stand out in the light of a headlamp; we could sometimes spot them from as far as ~20 m. If a green-coloured individual is easier to detect than a brown-coloured one, and if some individuals have not yet assumed the green ‘night colour’ by the time our surveys started, our detection rate could increase as more and more individuals turn green as the evening progresses. Green anoles that have been moved from light to darkness attain complete pallor in 7–38 min (Kleinholz 1938), but we lack information on what level of twilight an anole considers ‘darkness’. Finally, if a person that is getting tired alters his/her search pattern, detections may increase without the anole ‘availability’ (for detection) changing. For example, if tired observers tend to forego far-away perches and look closer, and if the (relatively small) anoles are easier to detect when sleeping at closer perches, the shifted focus might render a higher number of anole sightings.

We also notice that the temporal change in anole detections appeared to be more pronounced for some observers than for others, suggesting an interaction effect not evident for the other taxa (Fig. 1).

We could not verify statistically any temporal change in gecko sightings, but the tendency of a decline during the night mimics the small decline in emerald tree skink sightings. Elsewhere, the circadian rhythm of the common house gecko (Hemidactylus frenatus) appears to be responsible for its reduced detectability late in the evening (Rodda et al. 2015). Thus, it is likely observers’ gecko detection skills did not decline throughout the evening.

A different pattern emerged for small mammals (rats, shrews and mice pooled), with a strong and statistically verified sighting decline as the evening progressed. These species are large and actively moving, so they should be more conspicuous and easily detected than the lizard taxa. Therefore, we interpret the mammal sighting decline as mostly caused by their circadian rhythm, not decreased observer vigilance.

Cumulative fatigue, different from sleepiness, is the potential exhaustion and/or boredom caused by searching the same area five or six nights per week, for several weeks in a row. For a rapid response effort like this, the lack of rewards in the form of brown treesnake captures – not just the realised lack, but also knowing the chances you will detect a snake are low – might also cause a gradual drop in concentration and focus (cf. Henke 1998). We asked the observers to record all potential brown treesnake prey items seen, mostly because we wanted to characterise the prey populations in the (presumed) absence of snakes to better understand the snake’s ecosystem impact, should an introduced population become established (cf. Fritts and Rodda 1998). However, we also believe that in the absence of ‘snake rewards’, offering other kinds of rewards (documented sightings of lizards, mammals, birds) help motivate observers to remain alert. This multiple-target surveying means widening one’s search image, which might potentially divert focus from a mental search image for the main target (Henke 1998). We believe this potentially negative impact on snake search image to be less of an issue than the assumed loss of alertness if observers are asked to ignore everything but snakes (that may not be present at the site).

We found no long-term ‘cumulative deployment fatigue’ (Fig. 2), although this conclusion is confounded by the unequal durations of breaks in the search–night series that different observers had.

We verified that many observers were significantly biased (compared with the pool of peer observers) in one or more (but in no case all) taxa they detected. Notably, for all 11 observers showing significant biases for or against both emerald tree skinks and green anoles, those biases were in opposite directions (Table 1). Observers used different headlamp models; those using lamps with a longer reach may have detected large and shiny emerald tree skinks perched in denuded, typhoon-damaged tree tops further away, and thus be positively skink-biased (anoles are smaller and more difficult to see far away). We did not address headlamp effects because they were almost perfectly collinear with observers (i.e. few observers alternated between different lamp types, therefore observer and lamp effects are confounded), and we had no desire to brand individuals as ‘good’ or ‘poor’ observers. However, we have previously shown that headlamp power and beam width may affect the spatial pattern of detected animals (Lardner et al. 2007). The apparent trade-off between detecting skinks and anoles may be due to a difficulty in maintaining multiple search images, or insufficient time to carefully scan substrates both close and far from the transect.

Possibly suggestive of lamp types playing a role in the skink : anole bias variation is that deviations in the ratio of small mammals : geckos were less extreme (ratios not presented in this manuscript). We believe that most mammals and geckos were seen in rather close vicinity to the observers, in which case the reach of the headlamp would cause less taxon bias.

Temporal and spatial differences in vegetation structure may affect visibility (and detectability) of animal taxa differentially, depending on their microhabitat preferences. Similar surveys were conducted on Saipan in 2009 (effort Σ = 50.7 km; J. Stanford, pers. comm.), but in a habitat that had not been subjected to recent typhoon damage. Sighting rates of the four taxa (taxa groups) in 2009 compared with 2016 (Table 1) agree with a species-by-vegetation interaction. The numbers of individuals km^–1 of transect were very similar across years (2009 : 2016) for geckos (5.2 : 5.2) and small mammals (9.4 : 8.9), but almost 12-fold higher post-typhoon for emerald tree skink (2.7 : 32.0) and more than four times higher for green anoles (3.4 : 14.9).

One of the observers may have mistakenly identified most or all green lizards as green anoles through part (or much) of the survey, when in reality some of them were emerald tree skinks. This is seen in Fig. 3 as the lowest Ls value and the highest Ac value. We recognise that occasional errors are bound to be present in a dataset encompassing >20 000 observations by 29 persons with varying backgrounds and experience. However, we notice that many of the proportions that deviate strongly from the observer pool means in Fig. 3 were associated with observers highly experienced in these taxa.

Some observers saw consistently more animals than others, regardless of the taxonomic group in focus (Fig. 4). We did not analyse potential sources of those ‘skill’ differences but merely note that on average, the observer’s affiliation and usual duties appeared to have little or no impact on ‘skill’ in detecting lizards and mammals (data not shown). Most of the CNMI participants had been trained in visual surveying for brown treesnakes while attending one or more rapid response training courses in Guam (Stanford and Rodda 2007). Collecting data on lizards and small mammals is part of those courses. Furthermore, among participants who had not taken such courses were experienced bird surveyors. Although it may be important to have basic surveying experience (Thompson and Mapstone 1997) and a good search image to maximise detection, some persons may be better suited than others to visually detect cryptic organisms (Rodda 1993).

The authors declare no conflicts of interest.