Visibility Enhancement of Scene Images Degraded by Foggy Weather Conditions with Deep Neural Networks

Abstract

Nowadays many camera-based advanced driver assistance systems (ADAS) have been introduced to assist the drivers and ensure their safety under various driving conditions. One of the problems faced by drivers is the faded scene visibility and lower contrast while driving in foggy conditions. In this paper, we present a novel approach to provide a solution to this problem by employing deep neural networks. We assume that the fog in an image can be mathematically modeled by an unknown complex function and we utilize the deep neural network to approximate the corresponding mathematical model for the fog. The advantages of our technique are as follows: (i) its real-time operation and (ii) being based on minimal input, that is, a single image, and exhibiting robustness/generalization for various unseen image data. Experiments carried out on various synthetic images indicate that our proposed technique has the abilities to approximate the corresponding fog function reasonably and remove it for better visibility and safety.

Academic Editor:Wei Wu

Department of Electronics & Computer Engineering, Hanyang University, Seoul 133-791, Republic of Korea

Received 19 March 2015; Revised 16 May 2015; Accepted 18 May 2015; 22 November 2015

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

Degraded visibility and lack of luminance in foggy weather pose a serious threat to the safety of drivers. These conditions increase the danger of vehicle collision and are a major cause of injuries and fatalities on roads covered with fog. Suspension of very fine droplets in the fog causes blocking and scattering of the light. This leads to less light reaching the driver's eye, lower contrast, and hence reduced visibility (Figure 1). To enhance visibility in bad weather is an area of high interest for the researchers. Various studies have been carried out to observe and model the effects of wide range of weather conditions on vision systems. The authors in [1-4] present such weather models. They analyze how scenes are affected by various weather conditions. These studies attempt to restore various attributes of a scene in an image degraded by bad weather conditions with the help of these weather models. A few researches, for example, [5, 6], suggest the use of polarizing filters to remove the effect of haze from the images. To realize the techniques described in [5, 6], two or more independent images are required to comply with the condition that the air light produces some measurable partial polarization in them. Weather removal algorithms proposed in [7] make use of a single image and some supplementary information provided by the user. A deep photo system is proposed in [8] based on the georegistration of the photographs. This georegistration enables access to large amount of geographic information system (GIS) data such as 3D models for cities, buildings, terrains, and structures; texture models; and depth maps. The authors of this study suggest the augmentation of this information with simple photographs to achieve various operations like dehazing, relighting, view synthesis, and expanding the field of view. Defogging/dehazing solutions relying entirely on single images are proposed in [9-12]. The single image visibility enhancement method in [9] develops an optimized cost function based on certain observations of images with and without weather effects. This cost function attempts to estimate the direct attenuation of the air light which is applied to the scene for visibility enhancement. In [10] the dark channel prior (DCP) is proposed for single image haze removal. This prior, based on some statistical observations of outdoor haze-free images, is combined with the haze imaging model to recover a haze-free image. The authors in [11] propose another single image dehazing method which is based on [10]. They improve the dehazing performance by modifying the DCP method by introducing a single median filter operation. They also study the effects of dehazing on image and video coding. In [12] restoration of hazy scenes from a single image is proposed by defining the image through a scene transmission model that takes into account the surface shading. A general survey of vision based vehicle detection methods for intelligent driver assistance systems is presented in [13]. Finally in [14, 15] the authors propose enhanced visibility algorithms that take fog effects into account and are particularly well suited for road images.

Figure 1: (a) Scene visibility without fog. (b) Scene visibility under foggy weather conditions.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

In this paper, we present a novel approach to improve the visibility of the scene in an image degraded by foggy weather conditions. We target enhanced visibility by generating an approximate model of the fog composition in the scene with a deep neural network [16]. This generalized model is then used for the restoration of scene quality in the image. Our proposed method performs this recovery of image scene in real time and it does not require any additional information. The proposed method is robust as it achieves good result for a large set of unseen foggy images.

2. Defogging by Deep Neural Network

2.1. Artificial Neural Networks

Artificial neural networks [17-19] are complex mathematical systems which tend to electronically simulate the working of a biological nervous system. These networks are made up of a large number of very simple computational units called the artificial neurons. The number of artificial neurons in a network can range from a few hundred to a several thousand artificial neurons. A neural network is formed by interconnecting these hundreds and thousands of artificial neurons in different topologies. One such interconnected network is shown in Figure 2. It is through this interconnection of simple computational elements that the high computational complexity of the neural networks is realized. Artificial neural networks are now being researched to provide solution to various problems like function approximation, regression analysis, time series prediction, classification, pattern recognition, optimization, decision making, data processing, filtering, and clustering and categorizing, to name but a few. The advent of immensely high CPU processing abilities has made it possible to realize more deep architectures [16, 20] for the neural networks and hence more mathematically complex models can be accomplished. A deep neural network has many hidden layers of neurons between the input layer and the output layer.

Figure 2: Deep neural network architecture for fog removal.

[figure omitted; refer to PDF]

2.2. Deep Neural Network for Visibility Enhancement

We present a deep neural network that accepts a foggy image as an input, models the corresponding fog composition in the scene, and produces a defogged version of the scene in the image as an output. The architecture of the proposed deep neural network for image defogging is shown in Figure 2. The network consists of an input layer, an output layer, and [figure omitted; refer to PDF] number of hidden layers sandwiched in between the two. The multiple hidden layers of the deep neural network are advantageous in realizing more efficient representation for the corresponding fog function. The learning problem consists of finding the optimal combination of weights so that the network function [figure omitted; refer to PDF] approximates a given function [figure omitted; refer to PDF] as closely as possible. The network learns this given function [figure omitted; refer to PDF] through some implicit examples. In this paper, the deep neural network is tailored to solve the visibility enhancement problem by training it on several foggy images and their corresponding original images (input-output pairs). In order to learn the generalized fog function and produce the corresponding defogged images the foggy images are divided into [figure omitted; refer to PDF] nonoverlapping blocks of size [figure omitted; refer to PDF] pixels. Each block is normalized to a range that optimizes the learning and results of the DNN. The DNN is customized for a one-dimensional vector input; therefore, the two-dimensional block is transformed into a one-dimensional vector [figure omitted; refer to PDF] by rasterization. This vector [figure omitted; refer to PDF] is presented as an input pattern to the network. The weights of the DNN are initialized randomly and the input pattern is forward propagated through the multiple hidden layers of the DNN to generate the propagation output activation. The activation function employed in our DNN is the hyperbolic tangent transfer function which is given by [figure omitted; refer to PDF] The tangent hyperbolic function produces the scaled output over the -1 to +1 closed range. This function shares many properties of the sigmoid function, but because the output space of this tangent function is broader, it may be more efficient for modeling complex nonlinear relations more abstractly. This input pattern [figure omitted; refer to PDF] produces an output that is different from the target output (original Image). The error function of the network is the combination of errors contributed by all the hidden nodes of the network and is given as [figure omitted; refer to PDF] The combination of weights which minimizes the error function is considered to be the solution of learning. In this paper, this optimum set of weights is achieved by the backpropagation algorithm. The backpropagation is an iterative gradient descent algorithm in which the output error signals are transmitted backwards from the output layer to each node in the hidden layer that immediately contributed to the output layer. This backward propagation is continued, layer by layer, until each node in the network has received an error signal that describes its relative contribution to the overall error. Once the error signal for each node has been determined, the errors are then used by the nodes to update the values for the weights of the network. This backpropagation of errors and the updating of weights continue until the value of error function becomes sufficiently small. The training of the network is stopped and the new input patterns are presented to the network. The trained network can approximate and remove the fog in these input patterns for better visibility. The flow chart for the visibility enhancement DNN is shown in Figure 3.

Figure 3: Flow chart for proposed visibility enhancement DNN.

[figure omitted; refer to PDF]

3. Experimental Results

Experiments were carried out on images obtained from the FRIDA database [14]. The FRIDA database consists of sets of synthetic images with and without fog. The evaluation of the proposed method is carried out on gray scale images. The size of each image was 640 × 480 pixels. The image data set was divided into training set images and the test set images. The learning of deep neural network for defogging purpose was carried out by the training set. Architectures based on multiple hidden layers (5 to 8) with various numbers of hidden nodes (16 to 128) were trained. The convergence of the network was checked on the basis of mean square error (MSE). Once the network learnt the generalized fog function, the results were applied to the test images. For evaluation purpose we show the results in Figures 4-10. In these figures the areas of interest are shown enclosed in rectangles. The proposed solution can be evaluated by analyzing these areas. Figure 4(a) shows the original scene in the image without fog, while Figure 4(b) shows the scene affected by fog. It can be seen clearly that the background in Figure 4(b) is faded and is mostly invisible. The tree enclosed in the rectangle in Figure 4(a) is barely visible in Figure 4(b). The major portion of the building (front) which is visible in Figure 4(a) is completely invisible in Figure 4(b). Also other sections of the building enclosed in rectangles in Figure 4(a) are invisible in Figure 4(b). Figure 4(c) represents the scene recovered by our deep neural network. The building that was invisible in the foggy version is recovered in the defogged version. The tree is visible and other areas enclosed in rectangles in the figure are also partially recovered by the network. Figures 5(a) and 5(b) show the original scene and the fogged scene, respectively. The front building which is clearly visible in the original scene that is Figure 5(a) is faded in the fogged scene, while the building behind it is not visible in the scene of Figure 5(b). The smaller palm tree is also invisible in the foggy version. The house and trees at the left-hand side of the foggy image are almost invisible. The traffic signs and some minor structures are missing in the foggy scene. In Figure 5(c), the fog is removed by the deep neural network. The faded front building has improved visibility in the recovered image. The building behind the front building which was not visible previously can be clearly seen in our recovered scene. The smaller palm tree is visible. The house and tree at the left-hand side of the image are also recovered successfully. The traffic signs and the minor structures that were invisible due to fog are also recovered to some extent in the defogged scene. Figure 6(a) shows the original scene, while Figure 6(b) represents the image with degraded visibility due to fog. The trees enclosed in the rectangles on the left-hand side of Figure 6(a) are barely visible in Figure 6(b). The cars enclosed in the rectangles in the original image are invisible in the degraded image. Also the structures/buildings enclosed in rectangles in Figure 6(a) are invisible in the degraded version. The same scene with enhanced visibility is presented in Figure 6(c). As can be seen, the trees which are barely visible in the foggy image have been recovered. The cars which were previously invisible can be spotted in the recovered image. Also the structures/buildings missing in the foggy image are retrieved by our DNN successfully. Figure 7(a) shows the original scene in the image without any fog and Figure 7(b) represents the scene with reduced visibility due to foggy conditions. Starting from the left-hand side of the image the tree which is visible in the original image is barely visible in the degraded image. The house and the tree adjacent to it are invisible in the degraded image. The palm tree is also missing in the degraded image. The pole, tree, and the rear view of the building on the right-hand side are severely degraded and are invisible in the foggy image. Figure 7(c) represents the scene recovered in the image by the DNN. Starting from the left-hand side the tree which is barely visible has been recovered in the enhanced version. The house and adjacent tree have also been retrieved in the reproduced image, although the details are missing but the structures are visible. The palm tree which was previously invisible is present in the recovered scene. The pole, tree, and rear portion of the building are also recovered and are clearly visible in the reproduced image. Figures 8(a), 8(b), and 8(c) show the original image, the image affected by foggy conditions and the image recovered by our DNN. Compared to Figure 8(a), in Figure 8(b), the background that is buildings, houses, trees, vegetation, and cars enclosed in the rectangular boxes is barely visible or invisible, while in Figure 8(c) our proposed DNN has recovered the above regions successfully. Figures 9(a) and 9(b) represent the original image and its foggy version, respectively. The houses at the end of the lane in the left-hand side of Figure 9(a) are almost invisible in Figure 9(b). The poles and traffic signals are also invisible. The building at the right-hand side is faded and the trees behind it are not visible due to fog. The cars in the scene are barely visible. In our recovered image that is Figure 9(c) the visibility of the scene is enhanced. The houses at the end of the lane are now quite visible. The poles and traffic signals are recovered from the foggy version. The visibility of the building at the right-hand side is improved and the trees behind it are now visible. The cars in the recovered image are more detectable than the foggy version. Figure 10(a) represents the original scene without fog. Figure 10(b) represents the image scene degraded by fog. The building at the right-hand side is faded due to fog and is barely visible. The tree behind the building at the right-hand side is invisible. The traffic sign is invisible in the foggy image, and a large section of buildings and the palm tree is also missing in the left-hand side of the foggy image. In Figure 10(c) that is the scene reproduced by the DNN the barely visible building on the right-hand side of the image is largely visible. The tree behind is also recovered. The traffic sign is also visible in the reproduced image. The rear section of buildings and the tree on left-hand side have also been recovered successfully by our DNN.

Figure 4: (a) Original scene without fog. (b) Scene visibility degraded under fog. (c) Visibility enhancement by deep neural network.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

(c) [figure omitted; refer to PDF]

Figure 5: (a) Original scene without fog. (b) Scene visibility degraded under fog. (c) Visibility enhancement by deep neural network.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

(c) [figure omitted; refer to PDF]

Figure 6: (a) Original scene without fog. (b) Scene visibility degraded under fog. (c) Visibility enhancement by deep neural network.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

(c) [figure omitted; refer to PDF]

Figure 7: (a) Original scene without fog. (b) Scene visibility degraded under fog. (c) Visibility enhancement by deep neural network.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

(c) [figure omitted; refer to PDF]

Figure 8: (a) Original scene without fog. (b) Scene visibility degraded under fog. (c) Visibility enhancement by deep neural network.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

(c) [figure omitted; refer to PDF]

Figure 9: (a) Original scene without fog. (b) Scene visibility degraded under fog. (c) Visibility enhancement by deep neural network.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

(c) [figure omitted; refer to PDF]

Figure 10: (a) Original scene without fog. (b) Scene visibility degraded under fog. (c) Visibility enhancement by deep neural network.

(a) [figure omitted; refer to PDF]

(b) [figure omitted; refer to PDF]

(c) [figure omitted; refer to PDF]

4. Conclusions

In this paper, we propose a novel approach for restoring the visibility of foggy images with the help of deep neural networks. This idea is an initial attempt to model the fog function for scene recovery with deep neural networks. The method recovers the scene structure simply with a single image input, operates in real time, and generalizes well. The restored images showed good recovery of the scene from the foggy images. In some of the images the proposed technique showed limitations in reproducing the exact details of some objects in the scene, but, nevertheless, they were fairly recognizable. This method can be tailored to be used in applications designed for driving assistant systems, intelligent vehicle systems, outdoor surveillance systems, and so forth. In future works we are going to use this idea to develop more advanced frameworks for weather removal vision systems based on deep neural networks.

Acknowledgment

This research was supported by the MSIP (Ministry of Science, ICT and Future Planning), Republic of Korea, under the project for Technical Development of Information Communication and Broadcasting, supervised by IITP (Institute of Information and Communications Technology Promotion) (IITP 2015-B0101-15-1377).

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

References

[1] S. G. Narasimhan, S. K. Nayar, "Vision and the atmosphere," International Journal of Computer Vision , vol. 48, no. 3, pp. 233-254, 2002.

[2] S. G. Narasimhan, S. K. Nayar, "Chromatic framework for vision in bad weather," in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition (CVPR' 2000), pp. 598-605, June 2000.

[3] S. G. Narasimhan, S. K. Nayar, "Contrast restoration of weather degraded images," IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 25, no. 6, pp. 713-724, 2003.

[4] S. K. Nayar, S. G. Narasimhan, "Vision in bad weather," in Proceedings of the 7th IEEE International Conference on Computer Vision (ICCV '99), vol. 2, pp. 820-827, September 1999.

[5] Y. Y. Schechner, S. G. Narasimhan, S. K. Nayar, "Instant dehazing of images using polarization," in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 325-332, December 2001.

[6] S. Shwartz, E. Namer, Y. Y. Schechner, "Blind haze separation," in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), vol. 2, pp. 1984-1991, June 2006.

[7] S. G. Narasimhan, S. K. Nayar, "Interactive deweathering of an image using physical models," in Proceedings of the IEEE Workshop Color and Photometric Methods in Computer Vision, in Conjunction with IEEE International Conference on Computer Vision, October 2003.

[8] J. Kopf, B. Neubert, B. Chen, M. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendaele, D. Lischinski, "Deep photo: model-based photograph enhancement and viewing," ACM Transactions on Graphics , vol. 27, no. 5, article 116, 2008.

[9] R. T. Tan, "Visibility in bad weather from a single image," in Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition (CVPR '08), June 2008.

[10] K. He, J. Sun, X. Tang, "Single image haze removal using dark channel prior," in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '09), pp. 1956-1963, IEEE, Miami, Fla, USA, June 2009.

[11] K. B. Gibson, D. T. Vo, T. Q . Nguyen, "An investigation of dehazing effects on image and video coding," IEEE Transactions on Image Processing , vol. 21, no. 2, pp. 662-673, 2012.

[12] R. Fattal, "Single image dehazing," ACM Transactions on Graphics , vol. 27, no. 3, pp. 988-992, 2008.

[13] Z. Sun, G. Bebis, R. Miller, "On-road vehicle detection: a review," IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 28, no. 5, pp. 694-711, 2006.

[14] J.-P. Tarel, N. Hautiere, A. Cord, D. Gruyer, H. Halmaoui, "Improved visibility of road scene images under heterogeneous fog," in Proceedings of the IEEE Intelligent Vehicles Symposium (IV '10), San Diego, Calif, USA, June 2010.

[15] N. Hautière, J.-P. Tarel, D. Aubert, "Mitigation of visibility loss for advanced camera-based driver assistance," IEEE Transactions on Intelligent Transportation Systems , vol. 11, no. 2, pp. 474-484, 2010.

[16] G. Hinton, L. Deng, D. Yu, G. Dahl, A.-R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition," IEEE Signal Processing Magazine , vol. 29, no. 6, pp. 82-97, 2012.

[17] S. O. Haykin Neural Networks and Learning Machines , Prentice Hall, 2008., 3rd.

[18] Y. S. Abu-Mostafa, M. M. Ismail, H.-T. Lin Learning from Data , of AMLBook, 2012.

[19] P. H. Sydenham, R. Thorn Handbook of Measuring System Design , vol. 3, John Wiley & Sons, Chichester, UK, 2005.

[20] Y. Bengio, "Learning deep architectures for AI," Foundations and Trends in Machine Learning , vol. 2, no. 1, pp. 1-27, 2009.