A Box-Based Method for Regularizing the Prediction of Semantic Segmentation of Building Facades

Abstract

Semantic segmentation of building facade images has enabled a lot of intelligent support for architectural research and practice in the last decade. However, the classifiers for semantic segmentation usually predict facade elements (e.g., windows) as graphics in irregular shapes. The non-smooth edges and hard-to-define shapes impede the further use of the predicted graphics. This study proposes a method to regularize the predicted graphics following the prior knowledge of composition principles of building facades. Specifically, we define four types of boxes for each predicted graphic, namely minimum circumscribed box (MCB), maximum inscribed box (MIB), candidate box (CB), and best overlapping box (BOB). Based on these boxes, a three-stage process, consisting of denoising, BOB finding, and BOB stacking, was established to regularize the predicted graphics of facade elements into basic rectilinear polygons. To compare the proposed and existing methods of graphic regularization, an experiment was conducted based on the predicted graphics of facade elements obtained from four pixel-wise annotated building facade datasets, Irregular Facades (IRFs), CMP Facade Database, ECP Paris, and ICG Graz50. The results demonstrate that the graphics regularized by our method align more closely with real facade elements in shape and edge. Moreover, our method avoids the prevalent issue of correctness degradation observed in existing methods. Compared with the predicted graphics, the average IoU and F1-score of our method-regularized graphics respectively increase by 0.001–0.017 and 0.000–0.012 across the datasets, while those of previous method-regularized graphics decrease by 0.002–0.021 and 0.002–0.015. The regularized graphics contribute to improving the precision and depth of semantic segmentation-based applications of building facades. They are also expected to be useful for the exploration of data mining on urban images in the future.