Real-time pollen monitoring using digital holography

Abstract

We present the first validation of the Swisens Poleno, currently the only operational automatic pollen monitoring system based on digital holography. The device provides in-flight images of all coarse aerosols, and here we develop a two-step classification algorithm that uses these images to identify a range of pollen taxa. Deterministic criteria based on the shape of the particle are applied to initially distinguish between intact pollen grains and other coarse particulate matter. This first level of discrimination identifies pollen with an accuracy of 96 $\mathrm{\%}$. Thereafter, individual pollen taxa are recognized using supervised learning techniques. The algorithm is trained using data obtained by inserting known pollen types into the device, and out of eight pollen taxa six can be identified with an accuracy of above 90 $\mathrm{\%}$. In addition to the ability to correctly identify aerosols, an automatic pollen monitoring system needs to be able to correctly determine particle concentrations. To further verify the device, controlled chamber experiments using polystyrene latex beads were performed. This provided reference aerosols with traceable particle size and number concentrations in order to ensure particle size and sampling volume were correctly characterized.

1 Introduction

The incidence of pollinosis and related diseases has increased considerably over the past decades, sparking growing research interest into aeroallergens and pollen monitoring. Among aeroallergens, pollen is the most important impacting approximately 20 $\mathrm{\%}$ of the population in Switzerland and other high-income countries . Most often, sensitized patients exposed to allergenic pollen experience symptoms of allergic rhinitis or hay fever, but exposure to pollen has also been shown to exacerbate the development of more severe diseases like asthma, all of which have significant effects on public health and the economy .

Beyond the issue of public health, the airborne transport of pollen plays a key role in ecosystem dynamics, with important implications for agriculture, forestry, and the geographic dispersion of plants . The relevance of pollen and other bioaerosols for atmospheric chemistry and physics has also been increasingly acknowledged since they represent a significant fraction of atmospheric particulate matter and have been shown to influence cloud formation and precipitation . Furthermore, in the context of climate change, pollen concentrations undergo fluctuations in terms of taxa, abundance, and seasonal trends. Pollen monitoring thus provides valuable information about the evolution of the local biosphere and its response to anthropogenic forcings such as pollutant emission or intensive urbanization. While still uncertain, some evidence shows that the combination of a globally warming climate and the perpetuation of contemporary human lifestyle is very likely to increase the prevalence, intensity, and related costs of pollen-related allergenic diseases in the coming decades .

Airborne pollen has been monitored since the mid-twentieth century in Switzerland and elsewhere in Europe , most commonly with Hirst-type samplers . These instruments continuously collect airborne particles on a rotating cylinder tape, which is then collected, and pollen particles are manually identified and counted using optical microscopy, typically on a weekly basis. Because this is such a time- and labour-intensive method, the spatial and temporal resolution of the measurements is severely limited. Another drawback of this type of sampler is the inevitable delay between the observations and their analysis (up to 9 d), which has important implications in terms of pollen forecasts. In particular, the availability of real-time data with high temporal resolution is a key step in the development of accurate forecasting models for atmospheric pollen transport . More accurate predictions would represent a tremendous asset for both the scheduling of patients' activities and the planning of their medical treatment.

To respond to the need for real-time pollen information, numerous partly or fully automated monitoring systems have been developed and investigated over the past decade, with some recently having reached an operational level. Among the existing devices on the market, two main categories of instruments can be identified in terms of the different techniques utilized, either microscope-based or in situ measurements . The former aim to automatize the microscopic analysis process, while the latter make use of air-flow cytometry measurements, avoiding the collection step and performing real-time particle-by-particle identification. In the category of air-flow cytometers, most existing devices rely on fluorescence and elastic light-scattering measurements combined with machine-learning algorithms to identify and quantify airborne pollen concentrations. Some of these systems have already shown promising results and are currently tested in different European countries . Automatic pollen monitoring is part of a broader field of research on automatic bioaerosol monitoring , which was the object of a recent review article .

In this paper we evaluated a new automated pollen monitoring system based on air-flow cytometry, the Swisens Poleno. This device captures holographic images of each airborne particle in addition to measurements of optical properties such as fluorescence intensity, lifetime, and elastic light scattering. Here, we focus on the use of digital holography for online pollen monitoring since this technique allows a certain degree of visual identification of pollen taxa. We use a combination of classical image analysis and a neural network algorithm to assess the performance of the instrument in terms of pollen identification compared to manually classified calibration sets. Aerosol sampling, particle sizing, and counting performance are evaluated using a reference particle counter at the Swiss Federal Institute of Metrology, METAS .

In the following section, the Swisens Poleno air-flow cytometer is presented as well as the methodology used for the data analysis. Thereafter, the performance achieved in pollen identification and counting using holographic images from the device is shown. Although the focus of the present paper is on the use of digital holography to identify pollen grains, a validation of the output of the fluorescence using standard particles is also performed. Finally, the significance of the results for pollen monitoring are discussed, and an overview of the future perspectives for this new technology is provided.

2 Materials and methods

2.1 Swisens Poleno

We used the first unit of the commercially available Poleno device developed by Swisens AG (Switzerland). The device provides in-flight measurement of particle shape, size, and fluorescence using various light sources and detectors. The schematic structure of the device is presented in Fig. . Laser light scattering triggers the measurement, together with providing a first estimation of particle size, velocity, and alignment by combining the information of two trigger lasers. Following the trigger, two focused images at 90 ${}^{\circ }$ from each other are reconstructed using digital holography as in , and UV-induced fluorescence produces information regarding the particle composition. UV-induced fluorescence lifetime and spectra are measured at three different excitation wavelengths (280, 365, and 405 $\mathrm{nm}$) using five measurement emission windows between $\mathrm{320}$ and 720 $\mathrm{nm}$. Finally, a measurement of the time-resolved optical polarization characteristics of the particle is acquired before it exits the device.

2.2 Calibration dataset

A large ($>\mathrm{750}$ particles per pollen taxon) calibration dataset was collected for eight different pollen taxa using online measurements from the Swisens Poleno device. The taxa were chosen to present a good range of particle size – from small nettle pollen grains through to large pine pollen grains – and morphology. Note that the list includes taxa relevant for pollen allergies such as two different Betulaceae, Dactylis glomerata as a proxy for grasses, and ash. These samples were used to train a machine learning algorithm applied to identify the different pollen taxa. Only one particle type was calibrated at a time, allowing the data points to be labelled directly, although dirt, debris, and agglomerates needed to be eliminated from the dataset manually through visual inspection of the holographic images. To generate a large number of events without saturating the detector, pollen samples were continuously aerosolized using sound waves in a closed chamber around the detector inlet. Figure  shows examples of the reconstructed images generated for the calibration dataset. The pollen identification presented here is based just on these reconstructed images since they are expected to contain enough relevant morphological information to permit sufficient identification of the taxa of interest. Fluorescence and lifetime measurements are expected to be pertinent for extending the scope of the device to characterize other bioaerosols (e.g. spores) and pollutants. The dataset obtained includes 12 234 pollen grains (two images per grain) and is summarized in Table ; 80 $\mathrm{\%}$ of this dataset was used for algorithm calibration and 20 $\mathrm{\%}$ for validation purposes. The images are greyscale and have a resolution of 200 pixels $\times \mathrm{200}$ pixels. Each pixel represents a 0.56 $\mathrm{\mu }\mathrm{m}$ by 0.56 $\mathrm{\mu }\mathrm{m}$ physical domain.

Figure 2

Reconstructed holographic images from the Swisens Poleno for different pollen taxa: 1. Ambrosia artemisiifolia, 2. Corylus avellana, 3. Dactylis glomerata, 4. Fagus sylvatica, 5. Fraxinus excelsior, 6. Pinus sylvestris, 7. Quercus robur, and 8. Urtica dioica.

[Figure omitted. See PDF]

Table 1

List of pollen taxa used to train the classification algorithm, including the number of events used for training and validation of the machine learning algorithm for each taxa. Note that all pollen were in a dry state.

Common name	Taxa (Latin)	Supplier	# Training events	# Validation events
Ragweed	Ambrosia artemisiifolia	Bonapol	1063	266
Hazel	Corylus avellana	Bonapol	1156	289
Grasses	Dactylis glomerata	Bonapol	602	151
Beech	Fagus sylvatica	Allergon	859	215
Ash	Fraxinus excelsior	Allergon	826	207
Pine	Pinus sylvestris	Bonapol	3601	901
Oak	Quercus robur	Bonapol	775	194
Nettle	Urtica dioica	Bonapol	903	226

2.3 Shape analysis for pollen detection

A large range of coarse aerosol particles were seen in the events recorded by the Swisens Poleno. To provide a clean dataset to the pollen classification algorithm, the pollen particles needed to be discriminated from all others. In principle this can be done by applying thresholds to the confidence estimates provided by deep-learning pollen-classification algorithms; however, this simple method did not yield the required level of accuracy. An additional step was therefore implemented in the algorithm (thus becoming a two-step classifier), using shape analysis to discriminate between pollen and non-pollen particles prior to applying the full pollen classification.

In general, unbroken biological particles tend to have a smooth, convex shape, while dust, debris, or other nonbiological particles have rougher, more chaotic shapes (see, for example, Fig. ). Two deterministic image analysis routines were developed and evaluated to select the best available method for distinguishing pollen from other detected particles. Both use the contour of each particle, which is extracted from the reconstructed holographic images in three steps: (1) pixels are separated into two classes using the Otsu binarization algorithm , which is based on a dynamic intensity threshold; (2) the largest cluster corresponding to the particle of interest is then identified; and (3) a convolution operation extracts the contour of the particle.

The first routine for biological particle identification uses the OpenCV2 library to fit (in a least-squares sense) an ellipse to each contour . As a feature for biological particle identification the fraction $f_{\mathrm{c}}$ of the contour located further than a certain distance from the fitted ellipse is considered (red pixels in Fig. ). For pollen grains, this value is typically low, while for more fragmented particles this fraction can reach up to 60 $\mathrm{\%}$ of the contour (0 $\mathrm{\%}$ and 46 $\mathrm{\%}$ respectively for the examples shown in Fig. ).

2.4 Pollen classification using deep learning

Developments in computer hardware have made it possible to perform efficient training of complex artificial, or “deep”, neural networks; their use in image recognition problems is the iconic application of deep learning. Mimicking the visual cortex, a series of so-called convolutional layers identify relevant patterns and concentrate the information diluted over a large image. Extracted features are then used as input for fully connected layers of artificial neurons, which combine the features to determine associated labels for each image. This technique is part of the family of supervised-learning algorithms; networks need to be trained using images for which the label is known. We used the open-source software library Keras with TensorFlow as computational back end to implement the pollen identification algorithm.

Figure 4

Vision model based on the VGG16 architecture as used here for pollen classification.

[Figure omitted. See PDF]

The model used to classify pollen grains is based on the VGG16 architecture , which has successfully been applied to a wide range of different image classification problems . The basic model is shown in Fig. , with the vision model being applied separately to the two orthogonal images and the output then being processed with two fully connected layers. This ensures that the model is able to use the information from both images. For the final layer, softmax activation is used to map the network output to a probability distribution . The predicted pollen label is determined by taking the most probable class. Note that probability information is also useful since the plausibility of the final classifications can be easily verified . Furthermore, although not carried out here, a threshold could be applied to the classification when performing operational measurements to retain only the pollen grains classified above a sufficient confidence level.

3 Results and discussion

3.1 Pollen identification

The discrimination between pollen and other coarse aerosols is evaluated in Fig.  in the form of a normalized confusion matrix for each of the two image analysis algorithms. Each line in Fig.  is normalized to 1 and the values along the diagonal provide the recall for each category.

3.2 Pollen classification

Once a particle has been identified as a pollen grain it needs to be classified into the right taxa. Using the convolutional neural network (CNN) described in Sect. , each airborne particle is assigned a taxa with a corresponding confidence level of prediction. Results from the classification model are presented as a normalized confusion matrix in Fig. . The sum of each line is normalized to 1, and the diagonal values indicate the recall for the different pollen taxa.

Overall the classification algorithm performs very well, with six of the eight pollen taxa being classified with an accuracy of over 90 $\mathrm{\%}$. The exceptions are Corylus, which is confused in 10 $\mathrm{\%}$ of cases with Fraxinus, and Dactylis, which is confused 22 $\mathrm{\%}$ of the time with Corylus. Note that in this regard the problem presented to the algorithm is somewhat artificial; Corylus and grass pollen are not likely to be simultaneously present in the atmosphere in concentrations relevant for pollen allergies. Nevertheless, a larger mix of pollen taxa is likely to be observed in reality, highlighting the need for further developments to the classification algorithm using a larger number of species and including fresh pollen. In this line, it will be essential to include birch in the identification algorithm. This may, however, prove to be challenging given the morphological similarities of the members of the Betulaceae family.

Figure 6

Normalized confusion matrix for the pollen taxa identification, the second step of the classification algorithm.

[Figure omitted. See PDF]

To better understand the functioning of the neural network, Fig.  presents activation heat maps of pollen particles. These show which parts of the image the network focuses on to make the taxa prediction; in our case, strongly on the particle shape. This is apparent in the heat maps (Fig. ), as the highest activation regions follow the outline of the pollen grains. This may appear to be an obvious result but confirms the validity of the CNN step of the classification algorithm and indicates that the predictions are based on a physical feature of the particle and not on some other information embedded in the images. This verification is essential, as differences in light intensity or the presence of dust on lenses could lead to discrimination between calibrations not based on pollen morphology but on artefacts.

Figure 7

Visualization of the areas on which the convolutional neural network for pollen classification focuses.

[Figure omitted. See PDF]

Figure 8

(a, b) Concentrations (5 and 10 $\mathrm{\mu }\mathrm{m}$ PSL spheres) scaled to the METAS reference measurements in UTC. (c, d) Comparison of fluorescence measurements. Solid lines are the reference fluorescence intensities measured by the Max Planck Institute of Chemistry presented in . Median measurements from the Poleno are shown with error bars (interquartile range). Each excitation wavelength is scaled individually (see text for details).

[Figure omitted. See PDF]

Although there are some limitations to the use of dry pollen for model training purposes, the performance obtained suggests that holography alone is sufficient to distinguish between different pollen taxon. Combined with the results of the previous section on pollen identification, we propose a two-step approach for operational pollen monitoring using digital holography, first applying classical image analysis to identify pollen and subsequently using deep learning to classify these particles into individual pollen taxa. As mentioned in Sect. , the identification algorithm provides a measure of confidence in addition to the predicted label. Note that raw results are presented in the confusion matrix (Fig. ); in an operational setup confidence thresholds could be used to increase precision further. Due to the large sampling of such an automatic system, a certain loss of particles from introducing confidence thresholds can be accepted without losing statistical significance of the sampling.

3.3 Reference particle counts and fluorescence observations

The focus of this study was to assess the performance of the Swisens Poleno in terms of pollen identification. While this is key, it is equally as important to accurately quantify airborne pollen concentrations. At present, this remains a difficult task since no method, standardized or other, exists to aerosolize a known quantity of a known pollen taxa. Pollen grains are both considerably larger than other, nonbiological aerosol particles and relatively fragile, so producing homogenized airborne concentrations is currently not possible with conventional techniques.

To assess the accuracy of the particle concentrations obtained with the Swisens Poleno, a measurement campaign was carried out at the Swiss Federal Institute of Metrology (METAS). The custom-made facility at METAS has been described in detail in . The aim was to compare the Poleno device with reference particle concentrations and fluorescence observations in a controlled calibration chamber using polystyrene latex (PSL) spheres. Different sizes, ranging from 0.5–20 $\mathrm{\mu }\mathrm{m}$, were tested along with three types of fluorescent PSL (blue 2.07 $\mathrm{\mu }\mathrm{m}$, plum purple 2.07 $\mathrm{\mu }\mathrm{m}$, and red 2.07 $\mathrm{\mu }\mathrm{m}$) to provide a first insight into the quality of the fluorescence measurements. For each size, the concentrations measured by the Poleno were compared to the reference concentrations for approximately 20 $\min$. The fluorescent PSL used here have been fully characterized by the Max Planck Institute for Chemistry (MPIC) for a large range of excitation wavelengths. Those corresponding to the Poleno excitation wavelengths have been reproduced in Fig.  and serve as a reference for the fluorescence measurements. Since fluorescence intensity is measured in arbitrary units ($\mathrm{a}.\mathrm{u}.$), the fluorescence measured by the Poleno (filled dots) is scaled to the MPIC reference values (solid lines) using the maximum for each of the five emission windows located between 335 and 700 $\mathrm{nm}$.

The results presented here are encouraging, both in terms of particle concentration and fluorescence measurements. The Poleno seems to follow the fluctuations in terms of particle concentration very well, with Pearson correlation values of 0.905 and 0.916 for the 5 and 10 $\mathrm{\mu }\mathrm{m}$ sizes respectively (see Fig. ). Similar results are observed for the other particle sizes tested (not shown), indicating that the Poleno measures the size of the certified PSL particles correctly. It is important to note, however, that the Poleno values have been scaled to the METAS values since the particle concentrator is size selective, with larger particles being better sampled. Once a size-scaling curve has been established it can be effectively applied to all future measurements, which is a significant improvement compared to the current practice of deriving scaling factors for automatic pollen monitors from Hirst-type measurements . The systematic analysis of the efficiency of the concentrator goes beyond the scope of this paper but will be described in future work. The reproducibility of the scaling factors obtained was verified by repeating the experiments with the 2 $\mathrm{\mu }\mathrm{m}$ particles three times.

Despite the fact that the Poleno does not measure a continuous fluorescence emission spectrum, Fig.  confirms that it already provides an insight into the shape of the spectra for the different excitation wavelengths. The Poleno fluorescence signals agree well with the offline reference measurements performed at MPIC for all five emission windows and combined with the holographic images, potentially provide the opportunity to extend the number of particle types that can be recognized (e.g. further pollen taxa, spores, or pollutants).

4 Towards operational pollen monitoring

The focus of this study was to assess the performance of the Swisens Poleno, the first operational automatic pollen monitoring system based on digital holography. The potential of using these in-flight images to classify pollen particles in real-time was shown for eight pollen taxa using a two-step classification algorithm. The first step distinguishes intact pollen grains from other coarse aerosol particles using a deterministic ellipse-fitting method, providing a 96 $\mathrm{\%}$ discrimination accuracy for pollen. Thereafter, individual pollen taxa are recognized using supervised learning techniques. The algorithm is trained using data obtained by inserting known pollen types into the device, and six out of eight pollen taxa can be identified with an accuracy of above 90 $\mathrm{\%}$.

The ability of the device to accurately count particles was tested against reference measurements in controlled chamber experiments using polystyrene latex spheres. This is a key aspect for any monitoring device that is to be used operationally and to date has not been accurately assessed. These tests, together with validation of the fluorescence measurements carried out in the same chamber, provide very promising results.

The holographic images open the possibility for a human expert to perform online training and improve the model through a feedback loop. This effectively means that falsely classified pollen are identified manually and put into the correct class, for the model to use in the next training phase. The same principle could potentially be applied when the device is deployed in a new region with different pollen taxa by creating new pollen classes. Since the Swisens Poleno measures 1 m${}^{\mathrm{3}}$ of air every 25 $\min$, such new datasets can be created relatively quickly.

Finally, while not included in this study, the use of the fluorescence observations may allow the identification of particles other than pollen, for example, spores or other pollutants. Although the use of holography is a clear novelty of the present work, development of the method to additionally include florescence would build upon pioneering work performed using other devices . This could lead to synergies with air pollution monitoring networks and be of significant benefit to other sectors, such as agriculture and forestry, where real-time information concerning the distribution of spores could lead to better crop management practices. Future work in this direction is being continued, as is the development of the machine learning algorithm to identify further pollen taxa.

Code and data availability

All data and algorithms presented in the paper are experimental and subject to further development. They are available for research purposes on request to the authors of the paper. Work is in progress to harmonize the algorithms and make them public together with the data via open-software and data repositories.

Author contributions

BCl, BCr, ES, and YZ designed the study. BCr, KV, KA, and FT carried out the METAS campaign. ES and YZ analysed all available data. ES, YZ, BCr, and FT prepared the paper with contributions from all other authors.

Competing interests

The authors declare that they have no conflict of interest. At the time of writing YZ was affiliated with the Lucerne University of Applied Sciences and Arts but has since been hired by Swisens, AG. In no way did this affect this publication.

Acknowledgements

METAS has received funding from the EMPIR Projects 16ENV07-Aeromet and 19ENV08-Aeromet II. The EMPIR programme is co-financed by the participating states and from the European Commission Horizon 2020 research and innovation programme. This work also contributes to the EUMETNET AutoPollen Programme.

Financial support

The experiments carried out at the METAS were performed with funding from the EMPIR projects (grant nos. 16ENV-07-Aeromet and 19ENV-08-Aeromet II).

Review statement

This paper was edited by Francis Pope and reviewed by two anonymous referees.