Confidence-Feature Fusion: A Novel Method for Fog Density Estimation in Object Detection Systems

Abstract

Foggy weather poses significant challenges to outdoor computer vision tasks, such as object detection, by degrading image quality and reducing algorithm reliability. In this paper, we present a novel model for estimating fog density in outdoor scenes, aiming to enhance object detection performance under varying foggy conditions. Using a support vector machine (SVM) classification framework, the proposed model categorizes unknown images into distinct fog density levels based on both global and local fog-relevant features. Key features such as entropy, contrast, and dark channel information are extracted to quantify the effects of fog on image clarity and object visibility. Moreover, we introduce an innovative region selection method tailored to images without detectable objects, ensuring robust feature extraction. Evaluation on synthetic datasets with varying fog densities demonstrates a classification accuracy of 85.8%, surpassing existing methods in terms of correlation coefficients and robustness. Beyond accurate fog density estimation, this approach provides valuable insights into the impact of fog on object detection, contributing to safer navigation in foggy environments.

1. Introduction

Perception in outdoor scenes is essential for computer vision tasks such as object detection and tracking. Most image processing algorithms assume that input images are free from distortions caused by factors like fog or haze, making defogging a necessary step in certain situations. Adverse environmental conditions, particularly poor weather, can significantly impair the performance of these algorithms. However, existing defogging methods often require manual intervention to determine the variable parameters. This makes the process of selecting optimal parameter combinations both time-consuming and inefficient.

Furthermore, foggy weather can impair drivers’ visibility and judgment, potentially leading to severe traffic accidents and failures in transportation systems. Although defogging algorithms have been extensively researched, there remains a lack of effort in estimating fog density in dynamic environments.

In foggy conditions, the optical beams reflected by objects are influenced by the presence of aerosols and water droplets as they travel to the lens, resulting in blurred or unevenly gray-covered output results in the original scenes. The first solution leverages human prior knowledge to estimate the parameters of the atmosphere scattering model, with features like the dark channel [1] and fog-line [2] priors being the most effective priors proposed by researchers. The second solution tries to enhance the visual effect of foggy images. Ancuti [3] uses a fusion-based strategy to combine good quality regions in two images derived from the original image. However, the adjustable parameters inherent in traditional methodologies pose significant challenges in handling diverse fog density scenarios. Conversely, deep learning methods exhibit their strengths in intricate situations, particularly when the dataset comprises extensive and adequate image collections. They even possess the capability to defog images with non-uniform fog coverage. Nevertheless, the neural network model is notoriously difficult to interpret.

Instead of defogging a single image, we estimate global fog density by studying the influence of fog on object detection algorithms. For traditional defogging methods, fog density estimation is a good way to decide the parameters of defogging algorithms adaptively and avoid problems like excessive or insufficient defogging. Defogging algorithms can improve transmission and airlight by combining fog density estimation methods. At the same time, it provides us with an approach for perceiving an unknown environment. Vehicles equipped with a driving assistance system that have fog density analyzing capabilities can prompt drivers to avoid traffic accidents in time. For special vehicles with multiple visual detectors, fog density is also a reliable basis for making alternative selections.

In this paper, we will focus on the impact of fog density on object detection tasks and quantify it as overall fog density without the help of extra devices. Based on the fact that object detection algorithms cannot be fine-tuned in the haze dataset to optimize object loss, object detection confidence scores and positions will be combined in a fog density estimation model. Observing that the probability density distribution of confidence scores changes as fog density increases, we determine the distribution to be representative of explicit fog density. The deviation between the object confidence distribution of an unknown image sequence and standard foggy datasets can be used while predicting fog density.

Our methodology is realized by framing the fog density estimation process as a series of classification tasks. The feature space incorporates the most pertinent fog-relevant attributes to enhance the precision of the support vector machine. As depicted in Figure 1, the initial step involves extracting selectively chosen features from three distinct dimensions: the distribution of the object confidence scores across the image sequence, globally significant fog-relevant features, and localized features situated around object regions or areas automatically identified by an image patch selection technique. Subsequently, a support vector machine employing a one-to-one strategy is trained to establish a correlation between these features and the corresponding output labels.

After deciding the best parameters for the support vector machine via a grid search, the prediction model is constructed. Finally, we extract features for the unknown image sequences and predict them using this prediction model. Our main contribution has two parts: one is combining the distribution of object detection results and fog-relevant features into an estimation model, and the other is estimating fog density by means of solving a multi-class problem.

The main contributions of this work are the following:

• We introduce a novel estimation methodology that categorizes unknown images into distinct fog density classes based on a rigorously constructed standard fog dataset, leveraging the prediction of global fog density across continuous image sequences.

• We introduce an innovative approach that integrates the distribution of object detection confidence scores with fog-relevant features into a fog density estimation model, significantly improving the accuracy of fog density classification.

• We demonstrate an effective multi-class support vector machine approach for fog density classification, achieving an accuracy of 85.8% on a test set, showcasing superior correlation coefficients and robustness compared to existing methods.

The remainder of this paper is structured as follows: Section 2 provides an overview of related work, Section 3 details the proposed methodology, Section 4 discusses the experimental results and evaluations, Section 5 discusses the limitations of our methods, and Section 6 concludes with a summary and future directions.

2. Related Works

Current research focuses on estimating the ‘visibility’ distance and perceptual fog density in foggy environments. In meteorological terms, the International Civil Aviation Organization (ICAO) [4] defines visibility as the maximum distance at which a black object on a bright background can be discerned. Utilizing the mathematical relationship between the Weber contrast and the atmospheric attenuation coefficient, which serves as a proxy for human visual perception, the visibility distance can be computed once the atmospheric attenuation coefficient has been measured using physical instrumentation. However, these instruments typically necessitate a fixed location and a substantial distance, rendering them unsuitable for mobile scenarios or for addressing variations in attenuation coefficients stemming from non-uniform fog densities. Pomerleau [5] estimates visibility by quantifying the contrast attenuation of consistent road features situated at varying distances ahead of the vehicle. Although this approach obviates the need for a stationary device, it necessitates the presence of road markings for feature extraction, thereby constraining its applicability. Ding et al. [6] propose a novel approach to jointly estimate atmospheric light and the scattering coefficient from foggy image sequences by solving a unified non-linear optimization problem, addressing the limitations of sequential estimation strategies prone to error propagation. Choi’s [7] prediction model relies heavily on quantifiable deviations in image features observed in both naturally occurring foggy and fog-free images. Nonetheless, the definitive formula for determining fog density incorporates attributes from both foggy and fog-free datasets, which implies that images varying in fog density can exert an influence on the statistical value. Furthermore, constructing an ideal foggy dataset poses a significant challenge. Another approach presented by Jie Chen [8] involves comparing the scattering coefficient against two reference scales. However, the paper does not provide specific, fixed reference scales, and comparison methods inherently face the challenge of establishing clear and unambiguous references. Except for estimating overall fog density for an image, some methods estimate the density of each scene point. Jiang [9] defines fog density by the transmission coefficient. By fitting the surrogate model between fog features and optical depth, Jiang estimates depth for every pixel and estimates the fog density of each scene point. In addition, Xie et al. [10] introduce SynFog, a synthetic fog dataset generated through an end-to-end simulation pipeline that considers the entire physically based imaging process, enhancing the realism of synthetic foggy images for defogging algorithm research.

Recently, deep learning-based methods have achieved significant progress in image dehazing. AOD-Net [11] reformulates the atmospheric scattering model as a neural network, predicting clean images directly. Similarly, DehazeNet [12] predicts the transmission map using a multi-layer convolutional network. More advanced architectures, such as GridDehazeNet [13], leverage grid-like structures and attention mechanisms to better capture spatial and channel dependencies in hazy images. Generative adversarial networks (GANs) [14] have also been applied to enhance dehazing performance, enabling more realistic and visually appealing results. Accurate estimation of fog density is crucial for both image dehazing and related applications. Traditional methods rely on statistical priors such as contrast or brightness to estimate fog density [15]. Advanced methods use feature extraction techniques and regression models to predict density levels, either as discrete categories or continuous values. Recent studies have explored deep learning for fog density estimation. Khan et al. [16] proposed a CNN-based model that predicts density maps for different fog levels. Other works have integrated fog density estimation into end-to-end image enhancement frameworks [17], enabling simultaneous dehazing and fog-level classification. Such methods have proven beneficial in autonomous driving, where dynamic fog estimation enhances perception and decision-making under varying weather conditions.

The YOLO (You Only Look Once) series of models have consistently pushed the boundaries of real-time object detection. The model we utilized is the YOLOv5l model, and this model is further fine-tuned on an RTTS dataset. YOLOv8 and YOLOv10, the latest iterations, have introduced several innovations to improve both speed and accuracy. YOLOv8 incorporates dynamic convolution and an anchor-free design, allowing the model to adapt more effectively to varying object scales and shapes [18]. This version also employs a novel bag of freebies for training, enhancing generalization without additional inference costs. YOLOv10 builds upon YOLOv8 by integrating transformer-based modules for better feature representation across multiple scales [19]. Additionally, the model uses advanced techniques such as tokenized feature aggregation and global attention mechanisms, enabling superior performance on challenging datasets. These improvements make YOLOv10 particularly suitable for applications involving complex weather conditions, such as foggy environments.

3. Methods

In this section, we present the core methodology for fog density prediction based on synthetic data and fog-relevant feature extraction. We first introduce the physical model used to simulate varying fog densities in images, which enables the generation of synthetic datasets that support robust model training. Then, we define a set of global and local fog-relevant features to capture changes in image characteristics under different fog densities. Finally, we utilize a multi-class support vector machine (SVM) to classify fog densities based on these features, achieving accurate and reliable predictions.

3.1. Fog Density Physical Model and Dataset Generation

A physical model simplified from Koschmieder’s law is widely used in defogging algorithms. If the radius of particles in the air is ${10}^{-2}~10\left(\mathsf{\mu }\mathrm{m}\right)$ and the concentration is ${10}^3~1\left({\mathrm{cm}}^{-3}\right)$, fog takes shape in outdoor scenes. These air particles cause an impact on the reflected optical beam. The extent of this impact is decided by the particle properties, radius, and shape and hence forms different densities of fog.

Heavier fog can cause worse visibility degradation. There are two mechanisms that contribute to this phenomenon. One is the radiance attenuated by the optical beam caused by the absorption of particles as the distance between the scene point and the camera increases. The other is light reflected by the sun, sky, and ground, which makes the atmosphere behave like a source of light and increases the illumination of the scene. Therefore, light entering the lens is composed of two parts, attenuated light and scattered light from other optical paths. They are combined at different ratios at different scene points:

(1)$\mathrm{I}\left(\mathrm{x}\right)=\mathrm{J}\left(\mathrm{x}\right)\mathrm{t}\left(\mathrm{x}\right)+\mathrm{A}\left(1-\mathrm{t}\left(\mathrm{x}\right)\right),$

(2)$\mathrm{t}\left(\mathrm{x}\right)={\mathrm{e}}^{-\mathsf{\beta }\mathrm{d}\left(\mathrm{x}\right)}.$

Figure 2 and the equations above show the formation model, where I(x) is the foggy image captured by the imaging system; J(x) is the corresponding clear image; x is the position of every pixel; t(x) is the ratio mentioned above, which is called the transmission function and depends on the depth of scene point d(x) and medium attenuation coefficient $\mathsf{\beta }$; and A is the horizon radiance, called atmospheric airlight.

The key point of applying a multi-classification model to estimate fog density is to construct a labeled dataset with adequate foggy images. However, it is difficult to obtain corresponding clear and foggy images for the same scene, and it is even harder to label them with reliable densities.

To solve this problem, we synthesize foggy images with different densities from clear images by controlling the elements in the physical model mentioned in Equation (1). The fog density can be changed by the overlying airlight, which is global airlight A and transmission t(x). In this paper, we change the exponent in transmission in Equation (2) and introduce a new parameter α, as presented in Equation (3):

(3)$\mathrm{t}\left(\mathrm{x}\right)={\mathrm{e}}^{-\mathsf{\beta }\mathrm{d}\left(\mathrm{x}\right)/\mathsf{\alpha }},$

This modification allows us to generate varied images without directly altering the parameters $\mathsf{\beta }$ or d(x).

The specific synthesis process for foggy images is carried out using the atmospheric scattering model, which is shown in Figure 3, where the depth map of the clear image is given by the clear image through the depth map estimation model [20].

3.2. Feature Extraction for Fog-Relevant Attributes

The number of objects that can be detected decreases as the fog density increases. Images with no objects easily appear. Combined with the deviation in confidence distribution among unknown image sequences, fog-relevant features are used to judge the fog density auxiliary. We extract these features from two aspects. Global features like entropy, global contrast, and the standard deviation of grayscale are calculated through all pixels. Local features can eliminate the interference factors of image contents while distinguishing fog densities. For images with objects, object positions can help concentrate on local areas. For images without objects, distribution estimation and object position are not available. A region selection method is proposed to extract local features for those images with no objects.

3.2.1. Global Fog-Relevant Features

Entropy is an index for measuring the degree of system dispersion. It decreases as the system tends toward being uniformly distributed. Image entropy can reflect the detailed information in an image. A larger entropy value means the image has more details, like edges or variations among different areas. As the fog gets heavier, detailed information fades, and this is our first fog-relevant feature:

(4)$\mathrm{H}=-\sum _{\mathrm{i}=1}^{255}{\mathrm{p}}_{\mathrm{i}}\log {\mathrm{p}}_{\mathrm{i}}$

where ${\mathrm{p}}_{\mathrm{i}}$ is the probability of the pixel intensity calculated by a histogram.

Introduced by Matkovic [21], the global contrast factor uses contrasts at various resolution levels in order to compute the overall contrast of an image. In Figure 4, the final contrast is combined with the weighted average of local contrasts for nine image resolutions. Weight factors are defined by conducting experiments on human eyes. The calculation is as follows:

(5)$\mathrm{GCF}={\sum }_{\mathrm{i}=1}^9{\mathrm{w}}_{\mathrm{i}}\times {\mathrm{C}}_{\mathrm{i}},$

where ${\mathrm{C}}_{\mathrm{i}}$ is the average contrast of resolution i, and ${\mathrm{w}}_{\mathrm{i}}$ is the weighting coefficient.

The standard deviation of grayscale represents the deviation from its mean. The existence of fog enlarges all the grayscale in an image. The standard deviation of an image becomes smaller as the fog becomes heavier. The calculation is as follows:

(6)$\mathrm{S}={\left({\displaystyle \frac{1}{\mathrm{N}-1}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{|{\mathrm{A}}_{\mathrm{i}}-\mathsf{\mu }|}^2\right)}^{{\displaystyle \frac{1}{2}}} , \mathsf{\mu }={\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{A}}_{\mathrm{i}},$

where $\mathrm{N}$ is the number of pixels, $\mu$ is the mean of the gray image, and ${\mathrm{A}}_{\mathrm{i}}$ is grayscale in every pixel position.

Figure 5 and Table 1 show the trends in global features as the fog density increases. Images from (a) to (d) have densities of {0, 1, 3, 4}, and images from (e) to (h) have densities of {0, 1, 3, 4}. For the same scene, the entropy, the standard deviation, and the global contrast factor are getting smaller when the fog gets heavier. For different scenes, the concrete feature values may not be the same, but the disparity between features is distinct.

3.2.2. Local Fog-Relevant Features

The Weber contrast is commonly used in cases where small features are present on a large uniform background, i.e., where the average luminance is approximately equal to the background luminance. The results of object detection algorithms give us a way to distinguish between the foreground and background.

To obtain the background area surrounding an object, we expand the boundaries of the foreground by increasing the height and width so that the background area is approximately 1.6 times the foreground area. This ensures that the background region is sufficiently large to extract features while minimizing the overlap between different background regions. As shown in Figure 4, the areas bounded by yellow lines represent the foreground, while areas within the purple boundaries indicate the background. The Weber contrast ${\mathrm{C}}_{\mathrm{W}}$ between the foreground and background is calculated as follows:

(7)${\mathrm{C}}_{\mathrm{W}}={\displaystyle \frac{\mathrm{I}-{\mathrm{I}}_{\mathrm{b}}}{{\mathrm{I}}_{\mathrm{b}}}},$

where $\mathrm{I}$ and ${\mathrm{I}}_{\mathrm{b}}$ denote the luminance of the foreground and background, respectively.

The dark channel prior is a widely used and effective method for defogging images. Due to the colorfulness of objects and the shading effects, clear images typically exhibit very low values in their dark channels. When fog is present, however, the dark channel values increase. This property makes the dark channel a reliable indicator of fog. To represent the dark channel feature as a single value, we calculate the mean dark channel value across all objects and use it as a fog-relevant feature:

(8)${\mathrm{J}}^{\mathrm{dark}}\left(\mathrm{x}\right)=\underset{\mathrm{y}\in \mathsf{\omega }}{\min }\underset{\mathrm{c}}{\min }\left({\mathrm{J}}^{\mathrm{c}}\left(\mathrm{y}\right)\right),$

(9)$\mathrm{darkC}={\displaystyle \frac{1}{\mathrm{N}}}\sum _{\mathrm{k}=1}^{\mathrm{N}}\mathrm{mean}\left({\mathrm{J}}^{\mathrm{dark}}\left(\mathrm{x}\right)\right)$

In Ketan Tang’s [22] investigation of fog-relevant features, the local maximum saturation and dark channel were found to be visually correlated with fog density. Color saturation, in particular, is negatively correlated with fog density. To extract this feature, we convert images from RGB to HSV color space and calculate the mean saturation in the object regions.

The local features of an image are extracted by calculating the mean of all objects for images with objects. In Figure 6, the local region is chosen for the clear image (a) and foggy image (e). Some regions with cars are selected to calculate the Weber contrast, dark channel, and saturation. The Weber contrast is smaller for foggy image (e). In image (c), the car areas are dark except for a white car, and in image (g), all these areas have large values in the dark channel. The local saturation of clear and foggy images is shown in images (d) and (h). It has the opposite behavior to the dark channel.

3.2.3. Area Selection for Feature Extraction Without Objects

Object position information is not usable when there is no object in the scene. To extract local features, we select the target region from large to small scales. As is shown in Figure 7, taking the dark channel as an example, first, we divide the original image into four patches using the same-sized square, and then every patch is divided into four smaller ones. This division is performed three times to form 64 small patches, from which we will select one patch of interest. Second, we choose the region with the minimum dark channel value as the target region from the first division, which is patch 1. We continue to select the minimum dark channel value in patch 1 of the first division, which is patch 3. Then, the patch with position (4, 1) in patch 3 of the second division is selected as the final region for feature extraction. We calculate the maximum saturation feature for an image in this way.

Figure 8 shows the extraction regions of dark channel and saturation features under different fog densities in two target-free scenes. The red regions represent the dark channel feature extraction areas, while the blue regions indicate the saturation feature extraction areas. It can be observed that the selection of feature regions is generally correct: the most prominent dark channel regions are located in areas with very low grayscale values, and the saturation regions are chosen in areas with relatively high saturation values. The feature values are shown in Table 2.

3.3. Multi-Class SVM in Fog Density Classification

As outlined in Section 3.2. on feature extraction for fog-relevant attributes, the features of an image are extracted on both global and local scales. Subsequently, the average fog-relevant features across all images within a sequence are determined and designated as the defining features of that image sequence. To attain a reliable distribution of object detection scores, the kernel density estimation algorithm is applied uniformly to all the objects present within the image sequence. The aforementioned features are collectively incorporated to constitute the feature space. The labels assigned during the synthesis of the foggy datasets are leveraged as corresponding sequence labels. The support vector machine model, being the most prevalent supervised machine learning model for two-class classification tasks, is selected for classifying fog densities due to its ability to discern between separate classes with a maximal margin of separation.

The original SVM can only be used to group two classes. In this paper, five fog densities need to be classified. Therefore, the one-to-one method is used, and two-group classification models for every two fog densities are trained. Figure 9 shows the methods for training the SVM model. The output density is the one that has the highest score in prediction.

Training the classification model is straightforward: the inputs are all the fog-relevant features extracted globally and locally and the distribution features, and the output corresponds to fog density.

The LIBSVM library developed by Chang CC et al. [23] is utilized to implement the support vector machine model. All the features are scaled to [−1, 1]. The radial basis function is chosen as the kernel function. The gamma and cost parameters are determined through a grid search and the cross-validation tools provided by library.

4. Experiments

In this section, we present a series of experiments designed to evaluate the effectiveness of the proposed fog density estimation model across various synthetic foggy datasets. The experiments aim to (1) analyze the impact of fog density on object detection performance, (2) assess the discriminative power of the selected fog-relevant features, and (3) compare the accuracy of our model against baseline methods in predicting fog density. Through comprehensive quantitative and qualitative analyses, we demonstrate how the proposed features and the model architecture enhance fog density estimation accuracy across different visibility conditions.

4.1. Experimental Setup

4.1.1. Dataset Preparation

The initial dataset is based on the RESIDE-beta OTS [24] dataset. This dataset contains 2061 real outdoor images from Beijing under real-time weather conditions, and it uses deep learning algorithms to reduce depth errors and potential visual forgery, including images with haze as well as clear images. To ensure the diversity and realism of the dataset, clear images are used to generate haze images by simulating scenes under different fog densities. By applying the methods mentioned in Section 3.1, the final dataset is divided into three parts. The first part includes 593 images with massive objects and a similar scene depth to estimate the distribution of object confidence scores in different fog densities. Part two includes 3000 images as a training set to extract features and train the prediction model. Part three includes 600 images as a test set to test the classification model.

4.1.2. Parameter Settings

Following Koschmieder’s show, we choose normalized airlight A(A∼N(0.8, 0.05)) and β = 2.0 in the physical model. Using clear images and the corresponding depth map, we can generate four datasets of different fog densities and label them as {1, 2, 3, 4} by choosing α = {2.0, 1.5, 1.0, 0.6}. The larger α causes smaller t and smaller fog density. Clear images are labeled with density 0.

4.2. Evaluation Metrics

In addition to common evaluation metrics such as precision and recall, another four metrics are used for evaluation: Spearman’s rank-order correlation coefficient (SROCC), Kendall’s rank correlation coefficient (KLCC), Pearson’s linear correlation coefficient (PLCC), and the root mean square error (RMSE). SROCC and KLCC measure the monotonic relationships between predicted and actual fog densities, with values closer to 1 indicating stronger rank-based correlation. PLCC evaluates the linear correlation between predicted and ground truth values, where a higher PLCC signifies a closer alignment. RMSE provides a measure of the prediction error magnitude, with lower values indicating better predictive accuracy by assessing the square root of the average squared differences between predicted and actual fog densities. Together, these metrics comprehensively evaluate the model’s correlation performance and prediction accuracy.

4.3. Results and Analysis

4.3.1. Effect of Fog Density on Object Detection

Although it is desired, object detecting algorithms cannot be optimized on degraded image datasets to adapt complex outdoor scenes. To illustrate this fact, we conducted an experiment on YOLOv5l. As shown in Figure 10, RTTS is a subset of RESIDE, like OTS, but it contains foggy images and labeled objects, making it suitable for transfer learning. The model we utilized is YOLOv5l, which is fine-tuned on the RTTS dataset. During training, we observed that the loss curves for box_loss (Figure 11a) tend to converge, while the object confidence loss (obj_loss, Figure 11b) oscillates without convergence. This suggests that the object confidence score does not significantly improve with further training. Although this indicates that the confidence score can be a valid feature, we acknowledge that its stability could be influenced by several factors, including the choice of dataset, the training setup, and the model’s architecture. In this experiment, we hypothesize that the dataset’s characteristics, particularly fog density and the variability in object appearance, may limit the further optimization of confidence scores. Therefore, we recommend further investigation into the impact of different datasets, training strategies, and network architectures on the stability of confidence scores, as this result may not hold universally across all settings. Additionally, this experiment supports the notion that, in foggy environments, information recovery and feature extraction require more than just training on degraded datasets, as models trained on such data alone may fail to capture the full complexity of object detection tasks in real-world foggy conditions.

The variation is apparent after applying object detecting algorithms to images with different fog densities for the same scene. To demonstrate how fog density influences the object detection task, we utilized part 1 of dataset, which has sufficient objects in each scene and a uniform scene depth across all scenes. Figure 12 shows the detection results for two scenes with fog density {0, 2, 4}. The confidence scores of all the detected objects are shown in Figure 13. Important features like texture, colorfulness, and contrast disappear, which causes the number of objects that can be detected and the confidence score to decrease as the fog gets heavier.

In Figure 13, the 0 confidence scores observed for certain objects, such as bicycles or people, indicate detection failure under challenging conditions, particularly in scenes with high fog density. These failures reflect the limitations of the detection model under extreme visibility constraints. This finding emphasizes the importance of incorporating robust detection mechanisms or additional preprocessing steps, such as image defogging, to mitigate the impact of severe weather conditions. In the second scene, several objects face the issues of occlusion and being too far away. As a result, there are instances where the detector fails to function when the fog concentration is high. In fact, these objects are also extremely challenging for humans to detect. Although most situations align with expectations (i.e., the confidence level decreases as the fog concentration increases), there is an anomalous data point with the motorbike in the first scene. We suggest that the reason for this occurrence is that the high fog concentration blurs other complex background information in the scene, causing the detection algorithm to potentially misidentify the motorbike as a salient object, thus assigning a higher confidence level.

When the light conditions are good, there are still other factors, like scale variations or occlusions, that reduce the confidence level. We recommend using robust detection algorithms because they have been designed to handle these problems and no effort has been made to deal with foggy environments. The object detection algorithm we chose here is YOLOv4 [25], and we used pre-trained network weights, which have an accuracy of 65.7% [email protected] (43.5% [email protected]:0.95), on the MS COCO [26] dataset for the following parameters: width = 608, height = 608. There is no restriction on choosing object detection algorithms; one-stage and two-stage methods are both available.

The choice of YOLOv4 over YOLOv5l was driven by the need for better detection precision in this specific experimental setup. Although YOLOv5l offers advantages in terms of speed and lightweight architecture, YOLOv4 demonstrates superior performance in handling dense foggy scenes during preliminary evaluations. Its ability to balance detection accuracy and robustness helps minimize the influence of detection errors on the experimental results, ensuring the model’s evaluation focuses more on fog density estimation. Furthermore, YOLOv4’s well-optimized feature extraction pipeline and strong performance on challenging datasets like MS COCO made it an ideal choice for mitigating the effects of occlusions and scale variations in this study.

After applying object detection tasks to part 1 of the dataset and the corresponding foggy datasets, we can see the different output results of five densities. As illustrated in Figure 14, the number of objects decreases when fog gets heavier in the same scene set. In total, {9843, 6232, 5329, 4176, 2891} objects are detected for five datasets. The existence of fog causes a deterioration in contrast, colorfulness, and texture, which are significant features for identifying objects. However, relying solely on counting the number of objects in a single image is insufficient for accurately verifying its density. The quantity of objects may vary significantly across different scenes. Consequently, we employ confidence scores derived from object detection tasks, which are relatively immune to scene variations, to assess the probability distribution of these scores.

Object confidence reflects the reliability of the algorithm while detecting an object in foggy weather. We use kernel density estimation to obtain object confidence distribution considering that we have no prior to estimate this distribution:

(10)${\widehat{f}}_{\mathrm{h}}\left(\mathrm{x}\right)={\displaystyle \frac{1}{\mathrm{nh}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}\mathsf{\varphi }\left({\displaystyle \frac{\mathrm{x} -{\mathrm{x}}_{\mathrm{i}}}{\mathrm{h}}}\right),$

(11)$\mathsf{\varphi }\left(\mathrm{u}\right)={\displaystyle \frac{1}{\sqrt{2\mathsf{\pi }}}}{\mathrm{e}}^{-{\mathrm{u}}^2/2}.$

Equation (10) estimates the distribution of n samples, where ${\mathrm{x}}_1,{\mathrm{x}}_2,\dots ,{\mathrm{x}}_{\mathrm{n}}$ is a univariate independent and identically distributed sample drawn from the distribution of confidence with an unknown density f at any given point x, ϕ is the standard normal density function, and h is the bandwidth of the Gaussian kernel; h is auto-selected using smoothing techniques.

As the fog intensity increases in Figure 14, the confidence probability distribution undergoes a corresponding variation, analogous to the alteration in the quantity of objects. Notably, discernible differences in the distribution curves are evident, particularly in the region where the confidence level surpasses 80%. As the fog density escalates, the peaks of these curves undergo a contraction. To estimate the probability distribution of the synthesized datasets, we employ kernel density estimation with a Gaussian kernel. The confidence curves presented in Figure 14 are derived through the utilization of Equation (10).

Considering that the number of detected objects decreases as the fog density increases, the confidence scores from a single image are insufficient to accurately estimate the probability distribution. To overcome this limitation, we simulate the process of obtaining a sufficient number of object confidence scores by using multiple image frames taken in a short time period. However, since the dataset does not provide continuous image sequences, we create simulated sequences by randomly sampling images from the constructed dataset. Each simulated image sequence is comprised of a set of randomly selected images. For each sequence, we ensure that the number of objects detected is between 50 and 100. The lower limit of 50 objects ensures that there are enough detection instances to estimate the distribution, while the upper limit of 100 prevents the inclusion of sequences with an excessive number of objects. After collecting the confidence scores for the objects in each sequence, we calculate the probability distribution using kernel density estimation and then compute the area under the confidence curve where the confidence score is greater than 80%.

4.3.2. Fog Density Estimation Model Evaluation

To demonstrate the validity of the trained fog density estimation model, we predict fog density for image sequences sampled separately from the test set with five densities. In total, {100, 100, 100, 200, 100} image sequences are predicted, and the results are compared with real labels.

Figure 15 visually demonstrates the results for the image sequences extracted from the test set. Each column comprises a segment of an image sequence that belongs to the same prediction label. From left to right, the fog density escalates from 0 to 4. The initial row showcases images sourced from established foggy datasets, with densities ranging from 0 to 4, serving as a visual benchmark for the subsequent images. As the fog density intensifies, the prevalence of the overlying gray tone expands, and the intricate details gradually merge into the fog.

In Figure 16, a box–whisker plot is used to present the statistics of seven features for five fog densities after prediction using the test set. It plots the minimum, maximum, median, first quartile, and third quartile. From Figure 16, the local maximum saturation is not a good selection feature for the test set because the differences among the five densities are not clear and the features are all dispersed. Also, there are too many outliers. The global contrast, dark channel, and distribution of confidence scores are the ideal features. Local features like the dark channel and Weber contrast are more accumulative than global features when the fog density is great. This can help offset the adverse effect brought by the content of images.

For the test set, we present the model prediction results. The test set has five densities: {100, 100, 100, 200, 100}. A comparison between the predictions and the true labels and the classification results are shown by the confusion matrix in Figure 17. The horizontal axis represents the predicted class, and the vertical axis represents the true class. The dialog shows samples that are predicted correctly. The total accuracy for all the samples is up to 88.2%. For the five fog densities, the precision is {100%, 100%, 88.6%, 90.8%, 70.4%} and the recall is {100%, 87%, 84%, 79%, 100%}. TP, FP, and FN are the number of samples that predict positive to positive, negative to positive, and positive to negative, respectively. From Figure 17, we can see that precision and recall decrease as the density gets greater. Because the features are similar when the fog is very heavy, the model classification capability decreases. Moreover, our model tends to classify density to the adjacent greater density.

4.3.3. Comparison with Other Methods

Table 3 presents the results of the fog density estimation on the test set used in this paper, evaluated using the aforementioned four assessment metrics. BRISQUE [27] is the most commonly used no-reference image quality assessment method, and FADE [28] is a fog density assessment method. SROCC and KLCC metrics are used for the relative error of image assessment scores, primarily for evaluating the correlation of monotonicity. Since the FADE method assigns higher scores to greater fog density, and the methods used in this paper assign higher scores to better image quality, the larger the absolute values of these metrics, the more consistent the quality assessment method is with subjective evaluation. PLCC reflects the correlation of scores; the higher the PLCC is, the greater the correlation. RMES provides the error between the model’s predicted scores and the subjective scores, with lower errors being preferable. From the table, it can be observed how various assessment methods perform in evaluating the quality of foggy images. The method proposed in this paper is similar to the FADE method, whereas the commonly used image quality assessment algorithm BRISQUE performs poorly in quality assessment.

5. Discussion

The proposed method, while demonstrating significant improvements in controlled experiments, is not without limitations and uncertainties, particularly regarding its applicability to real-world scenes. A primary concern lies in its reliance on synthetic datasets for training and evaluation. Although these datasets are designed to replicate real-world conditions, they often fail to capture the full complexity of natural environments, such as unpredictable lighting, diverse object appearances, and environmental occlusions. This limitation may hinder the model’s generalization and lead to performance discrepancies when deployed in practical applications. Moreover, the method has been evaluated under specific conditions, such as predefined fog densities and object categories, leaving its robustness in drastically different operational contexts, such as extreme weather or novel environments, largely unexplored. These gaps in evaluation introduce uncertainties about the method’s scalability and generalization across diverse scenes.

Additionally, real-world deployment introduces further challenges, including sensor noise, dynamic interactions, and hardware limitations, which may significantly impact detection accuracy and computational efficiency. The reliance on pre-trained models like YOLOv4 [26] also brings potential biases, such as favoring specific object categories, and amplifies the risk of error propagation from the detection stage to downstream tasks. Although the method achieves enhanced detection accuracy, it may come at the cost of increased computational demands, posing a challenge for resource-constrained applications like real-time autonomous systems. These limitations highlight the need for future research focused on validating the method under diverse real-world conditions, integrating domain adaptation techniques to reduce dependency on synthetic data, and exploring lightweight models for greater efficiency. Such efforts will be crucial for bridging the gap between theoretical advancements and practical deployment while addressing ethical and operational considerations.

The choice of SVM in this study is motivated by its demonstrated robustness in high-dimensional spaces and small-to-moderate-sized datasets, which are key characteristics of our experimental setup. Additionally, SVM’s margin-based optimization aligns well with the requirements of this task, enabling the effective separation of classes in the feature space. Although alternative classifiers, such as random forests or neural networks, could be explored for comparison, these methods typically involve additional hyperparameter tuning and may require larger datasets for optimal performance. As such, SVM was chosen for its balance of simplicity, interpretability, and effectiveness in the given context. Future work could extend this study by exploring a broader range of classifiers to provide a more comprehensive analysis.

6. Conclusions

In this paper, we developed a comprehensive fog density estimation model tailored to object detection tasks in foggy environments. By combining object detection confidence scores with global and local fog-relevant features, our model successfully quantifies fog density using an SVM classifier. Experiments on synthetic fog datasets demonstrate the model’s robustness, with classification accuracy reaching 85.8% and favorable comparison with traditional methods. Although effective, our approach encounters limitations when scenes vary significantly in depth, potentially affecting fog density perception due to increased atmospheric light. Future work will address these depth-related variations and explore adaptive enhancements to further improve the model’s real-world applicability across diverse foggy conditions. We will also explore the effect of our method on more efficient and modern detectors.

Author Contributions

Methodology, Z.L., F.M. and L.Z.; dataset preparation, S.Z., F.M. and Z.F.; experiments, Z.L., S.Z. and L.Z.; original draft, Z.F., S.Z. and Z.L.; review and editing, S.Z. and L.Z.; visualization, L.Z., S.Z. and F.M. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

All the data mentioned in the paper are available through the corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Pipeline of the estimation model in this paper.

Figure 2. Simplified physical model of fog information.

Figure 3. Synthesis of four foggy datasets with α = {2.0, 1.5, 1.0, 0.6} from a clear dataset.

Figure 4. Global contrast factor and the Weber contrast calculation method. The yellow boxes are the foreground regions and the purple boxes are the background regions.

Figure 5. Visual effects of different fog concentrations in various scenes from images (a–h).

Figure 6. Extracting local features for images with objects. WC stands for Weber contrast, DC for dark channel, and S for saturation. (a) and (e) represent images with corresponding fog densities of 0 and 2, respectively; (b,f) show the extraction results of the Weber contrast; (c,g) show the extraction results of dark channel features; and (d,h) show the extraction results of saturation features.

Figure 7. Local feature extraction method for images without objects.

Figure 8. Extraction regions for dark channel and saturation features under different fog densities in two target-free scenes.

Figure 9. Training the SVM model to classify unknown image sequences.

Figure 10. RTTS dataset samples.

Figure 11. YOLOv5l fine-tuning process on the RTTS dataset. (a) represents box loss in the validation set; (b) represents object loss in the validation set.

Figure 12. Object detection results of two outdoor scenes. (a,d) are clear; (b,e) have a density of 2; (c,f) have a density of 4.

Figure 13. Object confidence of two scenes from Figure 12. (a) corresponds to the pictures in the first row; (b) corresponds to the pictures in the second row.

Figure 14. Object detection results for five datasets with fog densities of {0, 1, 2, 3, 4} synthesized from part 1 of the initial dataset. (a) denotes the difference in the number of targets; (b) denotes the differences in the probability density distribution of the target confidence level.

Figure 15. Prediction on a test set. Every column contains images of the same fog density as an output of the prediction model. From left to right, the fog densities are {0, 1, 2, 3, 4}.

Figure 16. Statistics of features for predicting outputs in the test set. All the features are scaled to [−1, 1]. SD stands for standard deviation, GCF for global contrast factor, DC for dark channel, WC for Weber contrast, and S for saturation.

Figure 17. Confusion matrix of the classification results.

Table 1

Results of global features for the different scenes in Figure 5.

Scenes	a	b	c	d	e	f	g	h
Entropy	7.3454	6.7220	5.6995	5.2639	7.8414	7.3404	6.1790	5.8581
Standard deviation	74.3098	44.7439	23.1754	14.1714	75.8730	48.6078	18.2677	16.1923
Global contrast	7.0592	3.6509	3.6585	2.2314	9.6159	3.2665	2.5195	2.8962

Table 2

The dark channel and saturation features extracted in Figure 8.

Features	a	b	c	d	e	f	g	h
Dark channel	0.1431	0.4089	0.5253	0.6251	0.0919	0.5073	0.6382	0.7410
Saturation	0.4550	0.3181	0.2306	0.1621	0.4036	0.1313	0.0771	0.0373

Table 3

Correlation coefficients and RMSE for the test set. The best results are highlighted in bold.

Metrics	SROCC	KLCC	PLCC	RMSE
FADE	0.8505	0.6194	0.7305	10.9638
BRISQUE	0.6156	0.5514	0.5067	13.3867
OURS	0.8723	0.7005	0.6716	8.9638

References

1. He, K.; Sun, J.; Tang, X. Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell.; 2010; 33, pp. 2341-2353. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/20820075]

2. Berman, D.; Treibitz, T.; Avidan, S. Non-local image dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognitionm; Las Vegas, NV, USA, 27–30 June 2016; pp. 1674-1682.

3. Ancuti, C.O.; Ancuti, C. Single image dehazing by multi-scale fusion. IEEE Trans. Image Process.; 2013; 22, pp. 3271-3282. [DOI: https://dx.doi.org/10.1109/TIP.2013.2262284] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23674449]

4. Moreno, I.; Jauregui-Sánchez, Y.; Avendaño-Alejo, M. Invisibility assessment: A visual perception approach. J. Opt. Soc. Am. A; 2014; 31, pp. 2244-2248. [DOI: https://dx.doi.org/10.1364/JOSAA.31.002244] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25401251]

5. Pomerleau, D. Visibility estimation from a moving vehicle using the ralph vision system. Proceedings of the Conference on Intelligent Transportation Systems; Boston, MA, USA, 12 November 1997; IEEE: Piscataway, NJ, USA, 1997; pp. 906-911.

6. Ding, Y.; Wallace, A.M.; Wang, S. Estimating Fog Parameters from an Image Sequence using Non-linear Optimisation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision; Waikoloa, HI, USA, 3–8 January 2024; pp. 1578-1586.

7. Choi, L.K.; You, J.; Bovik, A.C. Referenceless perceptual fog density prediction model. Human Vision and Electronic Imaging XIX; International Society for Optics and Photonics: Bellingham, WA, USA, 2014; Volume 9014, 90140H.

8. Chen, J.; Tan, C.H.; Chau, L.P. Haze density estimation via modeling of scattering coefficients of iso-depth regions. arXiv; 2018; arXiv: 1808.06207

9. Jiang, Y.; Sun, C.; Zhao, Y.; Yang, L. Fog density estimation and image defogging based on surrogate modeling for optical depth. IEEE Trans. Image Process.; 2017; 26, pp. 3397-3409. [DOI: https://dx.doi.org/10.1109/TIP.2017.2700720] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28475053]

10. Xie, Y.; Wei, H.; Liu, Z.; Wang, X.; Ji, X. SynFog: A Photo-realistic Synthetic Fog Dataset based on End-to-end Imaging Simulation for Advancing Real-World Defogging in Autonomous Driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Seattle, WA, USA, 16–22 June 2024; pp. 21763-21772.

11. Li, B.; Peng, X.; Wang, Z.; Xu, J.; Feng, D. AOD-Net: All-in-one dehazing network. Proceedings of the IEEE International Conference on Computer Vision (ICCV); Venice, Italy, 22–29 October 2017; pp. 4770-4778.

12. Cai, B.; Xu, X.; Jia, K.; Qing, C.; Tao, D. DehazeNet: An end-to-end system for single image haze removal. IEEE Trans. Image Process.; 2016; 25, pp. 5187-5198. [DOI: https://dx.doi.org/10.1109/TIP.2016.2598681]

13. Liu, X.; Ma, Y.; Shi, Z.; Chen, J. GridDehazeNet: Attention-based multi-scale network for image dehazing. Proceedings of the IEEE International Conference on Computer Vision (ICCV); Seoul, Republic of Korea, 27 October–2 November 2019; pp. 7314-7323.

14. Qu, Y.; Chen, Y.; Huang, J.; Xie, Y. Enhanced pix2pix dehazing network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Long Beach, CA, USA, 15–20 June 2019; pp. 8152-8160.

15. Hautière, N.; Tarel, J.-P.; Aubert, D.; Dumont, É. Blind contrast enhancement assessment by gradient ratioing at visible edges. Image Anal. Stereol.; 2008; 27, pp. 87-95. [DOI: https://dx.doi.org/10.5566/ias.v27.p87-95]

16. Khan, S.; Muhammad, K.; Mumtaz, S.; Baik, S.W.; de Albuquerque, V.H.C. Energy-Efficient Deep CNN for Smoke Detection in Foggy IoT Environment. IEEE Internet Things J.; 2019; 6, pp. 9237-9245. [DOI: https://dx.doi.org/10.1109/JIOT.2019.2896120]

17. Zhang, J.; Min, X.; Zhu, Y.; Zhai, G.; Zhou, J.; Yang, X.; Zhang, W. HazDesNet: An End-to-End Network for Haze Density Prediction. IEEE Trans. Intell. Transp. Syst.; 2022; 23, pp. 3087-3102. [DOI: https://dx.doi.org/10.1109/TITS.2020.3030673]

18. Redmon, J.; Farhadi, A. YOLOv8: Advanced real-time object detection. arXiv; 2022; arXiv: 2203.09800

19. Wang, A.; Chen, H.; Liu, L.; Chen, K.; Lin, Z.; Han, J.; Ding, G. YOLOv10: Real-Time End-to-End Object Detection. arXiv; 2024; arXiv: 2405.14458

20. Zhou, H.; Greenwood, D.; Taylor, S. Self-supervised monocular depth estimation with internal feature fusion. arXiv; 2021; arXiv: 2110.09482

21. Matkovic, K.; Neumann, L.; Neumann, A.; Psik, T.; Purgathofer, W. Global contrast factor-a new approach to image contrast. Comput. Aesthet.; 2005; 2005, pp. 159-168.

22. Tang, K.; Yang, J.; Wang, J. Investigating haze-relevant features in a learning framework for image dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Columbus, OH, USA, 23–28 June 2014; pp. 2995-3000.

23. Chang, C.C.; Lin, C.J. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol.; 2011; 2, pp. 27:1-27:27. Available online: http://www.csie.ntu.edu.tw/~cjlin/libsvm (accessed on 20 July 2024). [DOI: https://dx.doi.org/10.1145/1961189.1961199]

24. Li, B.; Ren, W.; Fu, D.; Tao, D.; Feng, D.; Zeng, W.; Wang, Z. Benchmarking single-image dehazing and beyond. IEEE Trans. Image Process.; 2018; 28, pp. 492-505. [DOI: https://dx.doi.org/10.1109/TIP.2018.2867951] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30176593]

25. Bochkovskiy, A.; Wang, C.Y.; Liao, H.Y.M. YOLOv4: Optimal speed and accuracy of object detection. arXiv; 2020; arXiv: 2004.10934

26. Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C.L. Microsoft coco: Common objects in context. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference; Zurich, Switzerland, 6–12 September 2014; Proceedings, Part V 13 Springer International Publishing: Berlin/Heidelberg, Germany, 2014; pp. 740-755.

27. Mittal, A.; Moorthy, A.K.; Bovik, A.C. No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process.; 2012; 21, pp. 4695-4708. [DOI: https://dx.doi.org/10.1109/TIP.2012.2214050] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/22910118]

28. Choi, L.K.; You, J.; Bovik, A.C. Referenceless prediction of perceptual fog density and perceptual image defogging. IEEE Trans. Image Process.; 2015; 24, pp. 3888-3901. [DOI: https://dx.doi.org/10.1109/TIP.2015.2456502]