Experimental Study on 3D Measurement Accuracy Detection of Low Altitude UAV for Repeated Observation of an Invariant Surface

Abstract

Low-altitude unmanned aerial vehicle (UAV) photogrammetry combined with structure-from-motion (SFM) algorithms is the latest technological approach to imaging 3D stereo constructions. At present, derivative products have been widely used in landslide monitoring, landscape evolution, glacier movement, volume measurement, and landscape change detection. However, there is still a lack of research into the accuracy of 3D data positioning based on the structure-from-motion of unmanned aerial vehicle (UAV-SFM) technology, itself, which can affect the measurable effectiveness of the results in further applications of this technological approach. In this paper, validation work was carried out for the DJI Phantom 4 RTK UAV, for earth observation data related to 3D positioning accuracy. First, a test plot with a relatively stable surface was selected for repeated flight imaging observations. Specifically, three repeated flights were performed on the test plot to obtain three sorties of images; the structure from motion and multi-view stereo (SFM-MVS) key technology was used to process and construct a 3D scene model, and based on this model the digital surface model (DSM) and digital orthophoto map (DOM) data of the same plot with repeated observations were obtained. In order to check the level of 3D measurement accuracy of the UAV technology itself, a window selection-based method was used to sample the point cloud set data from the three-sortie repeat observation 3D model. The DSM and DOM data obtained from three repeated flights over the surface invariant test plots were used to calculate the repeat observation 3D point errors, taking into account the general methodology of redundant observation error analysis for topographic surveys. At the same time, to further analyze the limits of the UAV measurement technique, possible under equivalent observation conditions with the same processing environment, a difference model (DOD) was constructed for the DSM data from three sorties, to deepen the overall characterization of the differences between the DSMs obtained from repeated observations. The results of the experimental study concluded that both the analysis of the 3D point set measurements based on window sampling and the accuracy evaluation using the difference model were generally able to achieve a centimeter level of planimetric accuracy and vertical accuracy. In addition, the accuracy of the surface-stabilized hardened ground was better, overall, than the accuracy of the non-hardened ground. The results of this paper not only probe the measurement limits of this type of UAV, but also provide a quantitative reference for the accurate control and setting of an acquisition scheme of the UAV-based SfM-MVS method for geomorphological data acquisition and 3D reconstruction.

1. Introduction

In recent years, fast 3D reconstruction algorithms using structures from motion and multi-view stereo (MVS) techniques, high-spatial-resolution imagery from ground or UAV, and a range of derivatives have been used in many applications. Compared to other digital measurement methods, the UAV-SfM-based method requires only a photographic overlap of the target object to obtain high-quality 3D terrain data quickly, without prior calibration of the camera and without high requirements for image size. Many researchers have referred to the generation of the DSM by drones in different application contexts. DSM is used directly as a derivative of UAV-SFM for landslide monitoring in geology and geotechnics [1,2]; in geomorphology, to study the temporal evolution of terrain [3,4,5,6]; in glaciology to estimate the volume change and movement of ice [7,8]; and in landscape monitoring and change detection [9] to support earth volume estimation [10], among others. Bi H’s team investigated the applicability of SfM photogrammetry in modelling the topography of fault zones, evaluating in detail the resolution and accuracy of SfM-derived topography against existing airborne LiDAR data. The results showed that the point cloud density generated by SfM photogrammetry was nearly 70-times higher than that generated by airborne LiDAR. The results show that the UAV-based SfM photogrammetry method can provide a cheap, effective, and flexible alternative to airborne LiDAR for topographic mapping of fault zones [11]. However, the accuracy of the UAV-SFM method derivatives depends on geometric and physical parameters. Examples include, image scale, ground sampling density, stereo base length to object distance ratio, aerial band overlap percentage, illumination, sensor quality, and algorithms for processing image data, such as the structure from motion, beam method levelling, image matching, and point cloud noise rejection. For derivative accuracy studies, many variables are involved and no single experimental study can fully encompass all quantities of variation [12,13]. A survey of the theory and methods of photogrammetric bundle adjustment, aimed primarily at potential implementers in the computer vision community, was presented by Triggs B et al. and found that bundle adjustment is a matter of refining visual reconstructions to produce common optimal structural and observational parameter estimates [14,15]. In their study, Özyeşil O et al [16]. stated that the structure-in-motion problem needs to recover the 3D structure of a stationary scene from a set of projection measurements by estimating the motion of the camera corresponding to those images. And after an section on earlier techniques for motion and structure estimation based on factorization, the team also provide a detailed description of some recent camera position estimation methods in the literature and then discuss important techniques for 3D structure recovery. Insufficient research on the accuracy of the integrated data results based on the UAV-SFM method has hindered the further application of the UAV-SFM method. A survey found that an increasing number of commercial UAV platforms support dual-frequency receivers for real-time kinematics (RTK), and that it is important to replace the error propagation of georeferencing techniques in this mode. In other words, if the global navigation satellite system (GNSS) receiver on board the UAV can receive centimeter-level positioning accuracy, then ground control points are no longer the only option for geo-alignment and transformation detection. Today, there has been a significant growth the in airborne photogrammetry of unmanned aerial vehicles (UAVs), due to the democratization of the use of UAVs in the civilian sector. This was used by Daakir M et al. for direct georeferencing of digital surface models (DSM), without relying on ground control point (GCP) measurements to initially probe the maximum accuracy achievable by a coupled camera and GPS receiver system [17]. For a comprehensive accuracy study of the data results of the UAV-SFM method, many researchers have used ground control point (GCP) data or terrestrial laser scanning (TLS)point clouds for accuracy comparison analysis with UAV-SFM data. Niels Anders et al [1]. demonstrated the effect of flight altitude and flight direction on the ground model and orthophoto resolution of low-altitude aerial photography from UAVs. Three flight heights and two different flight-trajectory orientations were tested. The results show that the UAS data products were able to clearly capture detailed morphology and vegetation structure. However, the difference in the vertical accuracy of the DSM was not significant with increasing flight altitude, and the altitude values were more sensitive to different flight path directions. The horizontal accuracy and lateral positioning of features in orthophotos does not vary with flight altitude and direction. Forlani G et al [18]. conducted an experimental study on the repeatability of UAV-generated DSM, by repeating the flight plan eight times at a distance of about 90 m from the ground and a ground sample distance (GSD) of 2 cm, and in three of the eight repetitions, elevation deviations of about 10 cm or more occurred. On the other hand, the addition of at least one GCP to the camera station succeeded in eliminating most of the bias in all the tests carried out, and the root mean square error (RMSE) at high altitudes reached 3 cm. Grottoli E et al [19]. explored a number of depth map filtering and point cloud refinement methods based on the commercial software Agisoft Metashape Pro, to determine the best method for building reliable digital surface models. Twelve different aerial photography-derived DSMs were used for validation. The experimental study found that the average vertical error of the different studied methods was about 1 m, and the average vertical error value of the DSM, with confidence values in the range 0–3 removed from the dense point cloud (DPC) and with the depth map filter disabled, was 0.93 m. Nikolov I A et al. addressed the fact that SfM has an inherent ambivalence for the scale of scanned objects and proposed a lightweight solution that calculates the absolute scale of 3D reconstructions by using only real-time motion (RTK) GPS [20]. Compared to other bespoke solutions that require multi-sensor fusion, the trials showed that the team’s proposed solution provides better results than the two state-of-the-art solutions. An inherent problem with SfM is the uncertainty of the scale of the reconstruction, due to the reliance on two-dimensional images. Nikolov I et al [21]. demonstrated how distance sensors can be used to calculate the scale of a reconstructed object. Furthermore, the authors demonstrated that uncertainty in the computed scale can be calculated, as well as how it depends on the accuracy of the sensors used. They investigated the accuracy of photogrammetric models and digital terrain model (DTM) generated by the DJI Phantom 4 RTK UAV platform in Agisoft Metashape. In the study, a DJI Phantom 4 RTK was used to acquire image data from two different angles for the same target in both vertical and oblique directions, and GCPs were introduced for accuracy assessment. The experimental results showed that the accuracy values for the oblique dataset with GCPs (3D RMSE = 0.025 m) and without GCPs (3D RMSE = 0.028 m) were relatively close. In contrast, for the vertically angled acquisition dataset, the location and number of GCPs had a greater impact on the accuracy (3D RMSE from 0.034 to 0.075 m). However, the introduction of tilted angle images improved the vertical accuracy of the vertical angle images (RMSE from 0.052 to 0.025 m). In addition, they compared the results of their study with those of the RTK model for the same dataset. Within a continuously operating reference station (CORS) distance of 15 km, there was no significant difference between the photogrammetric model and the DTM obtained with the RTK and post processing kinematic (PPK) tilt datasets without GCP. Li Wenda [22] and others, based on the UAV-SFM technique, investigated the effects of shooting height, shooting angle, and photo overlap on the measurement accuracy of geomorphological data, as well as on the effectiveness and generation time of regional 3D-reconstruction models. The experimental results showed that the shooting angle has different effects on horizontal accuracy and vertical accuracy. The more perpendicular the shooting angle is to the ground, the smaller the horizontal error and the larger the vertical error, and vice versa.

A comprehensive analysis of the current state of research at home and abroad concluded that the UAV-SFM algorithm has become increasingly mature, but the comprehensive accuracy of the data results obtained by the UAV-SFM method has not been studied sufficiently. This paper addresses DSM data obtained using a DJI Phantom 4 RTK consumer-grade UAV, through repeated observations of the same, unchanging area, and analyses the differences between DSMs, according to the general error analysis method of topographic surveying, combined with the sampling method of the window surface and the DoD-DSM model, which helps to measure the limits of UAV surveying technology for change detection, and provides a quantitative reference and technical support for the application and promotion of this type of UAV.

1.1. Data Collection

1.1.1. Study Area

The test area was selected to carry out accuracy verification tests on invariant terrain within a university in Kunming, Yunnan Province, with a longitude of 102°41′40.2″ and a latitude of 25°03′51.9″. The area contains rich terrain features, such as artificial structures, buildings, roads, and vegetation, etc. The terrain is flat and the vegetation cover is 70–80%, the measurement area was 63 * 66 m and the area was 4158 m². The area has a wide view and no satellite signal interference, which was conducive to the flight testing of the UAV. The location of the test area is shown in Figure 1.

1.1.2. UAV-Based Repeat Observation Data Acquisition Scheme

A DJI Phantom 4 RTK UAV was used to collect image data from the survey area. The aerial survey system mainly consists of the UAV flight platform, flight control system, and image sensors, etc. The positioning system uses a dual backup GNSS system and supports GNSS dynamic differential post-processing technology, making the aerial surveying operation flexible and efficient. The specific UAV platform parameters and camera parameters are shown in Table 1. The image processing working photo is a Lenovo Thinkvision with Inter (R) Xeon(R) CPU E5-2609 [email protected] GHz, 32 GB of memory, and Agisoft PhotoScan software for image data processing.

Our team tried to use a variety of different processing software when conducting the experimental study, based on the SFM method of 3D model construction. For example, pix4d, Agisoft PhotoScan, ContextCapture Center Master, and other software processing platforms. Through comprehensive comparison and analysis, we found that the completeness of the SFM 3D reconstruction data based on Agisoft Metashape software was relatively good and met the test requirements. One of the most recent is provided by the commercial software Agisoft Metashape (since version 1.6), previously known as Photoscan, which joins the other available opensource and commercial software tools. Since version 1.6, released in January 2020, Agisoft Metashape (a photogrammetric 3D modelling software based on the structure-from-motion technique) supports the processing of satellite imagery, in addition to standard digital images. This new option, integrated into the standard multi-view workflow, allows generating all the typical products, such as dense clouds, DSMs, and tiled polygonal models from satellite optical imagery. Specifically, Metashape can process panchromatic and multispectral satellite images, provided that sufficiently accurate rational polynomial coefficients (RPCs) are available for each image [23].

During the UAV data acquisition process, first, the DJI Pilot platform was used to route the survey area according to the actual surrounding geography of the survey area. Using the same planned route plan and based on the same altitude, three flights were performed to obtain three sets of image data to provide a dataset for the 3D error analysis of repeated photographic imaging observations of the UAV at a later stage [24].

The three sorties were named H40J1, H40J2, and H40J3, where H40 stands for the altitude of the aerial survey and Jx stands for the number of flight frames. The images were taken on 12 March 2021, in clear weather. The flight parameters were the same for all three sorties, the altitude was 40 m, the paralateral overlap and heading overlap were both 80%, and 65 images were acquired.

2. Research Methodology

2.1. Technical Framework for the Study

In order to carry out an experimental analysis of consumer-grade UAV accuracy detection for repeated observations of impermeable surfaces, the main technical framework of the study was identified as shown in Figure 2. Within this, the core methodological techniques used include the following three aspects: ① Construction of UAV DSM based on SFM-MVS; ② 3D point set sampling and accuracy analysis based on window surface; ③ Change detection and accuracy analysis based on DoD-DSM [25], [26].

As shown in Figure 2, first, the UAV repeat observation data acquisition was carried out, with three image sorties, namely H40J1, H40J2 and H40J3. Second, DSM was constructed using the SFM-MVS method for the acquired image data of the three sorties: DSM1, DSM2, and DSM3, and the scenes were visualized. The typical landforms were then decoded and the permeable surface I and impermeable surface II within the test area were circled in conjunction with the survey area DOM data. Finally, 3D point set sampling from the window surface with the DoD-DSM approach was used to analyze the 3D point errors of the three sorties of the aerial survey data.

2.2. DSM Construction for SFM-MVS

Three-dimensional reconstruction, based on the SFM algorithm, in UAV measurements is one of the hot issues in photogrammetry and computer stereo vision recovery research. The method relies on an UAV to acquire image data with a certain degree of overlap, to estimate the motion through the geometric relationship of the 2D image, and to determine the 3D point position by combining this with the beam method of leveling optimization. Finally, the scene is then densified with CMVS/PMVS, to recover the 3D model of the scene. The combination of SFM and MVS has been successfully used in UAV image processing to produce high resolution DSM and DOM data. The SFM sparse point cloud is constructed using a feature description and extraction algorithm, such as the scale-invariant feature transform (SIFT), combined with a K-s tree model to calculate the Euclidean distance between feature points and perform matching. The base or essential matrix is solved based on matched pairs of points, and then the 3D coordinates of the corresponding object square points are solved based on the triangular method. Given the existence of errors, the results are then optimized using the beam method of levelling. The process is to solve the optimal camera projection matrix and the 3D point coordinates by minimizing the error function, resulting in the camera’s positional attitude and sparse 3D point cloud.

The dense reconstruction of MVS is an encrypted 3D point cloud, based on the SFM sparse point cloud construction; as the sparse reconstructed point cloud is coarser and does not represent the 3D reconstruction information of the scene well. To ensure that the reconstructed 3D scene is more realistic, the MVS algorithm is able to reconstruct the 3D coordinates of almost every pixel, so that the density of the points is closer to the texture information that the image presents to us. The stereo reconstruction of the true 3D surface DSM is based on the MVS dense reconstruction, to recover the true 3D surface, based on depth maps with stereo images [27].

2.3. Window Sampling and DoD Differential Processing Techniques

According to the general error analysis method of topographic surveying, the accuracy assessment of point error analysis mainly refers to the measurement of the accuracy index between multiple periods of data, to ensure the correctness of the results of multiple periods or sorties of observation data, when the same target is measured at the same station and at different times.

The rating criteria depend mainly on the inter-compatibility between the different sortie data. Accuracy is mainly verified by the error in each dimensional point position, as shown in Equation (1).

(1)$\begin{array}{l}{\delta }_x=\pm \sqrt{\frac{{\displaystyle \sum _{i=1}^{\mathrm{n}}\Delta x_i{}^2}}{\mathrm{n}}},{\delta }_y=\pm \sqrt{\frac{{\displaystyle \sum _{i=1}^{\mathrm{n}}\Delta y_i{{}^2}_{{}^{}}}}{\mathrm{n}}},{\delta }_z=\pm \sqrt{\frac{{\displaystyle \sum _{i=1}^{\mathrm{n}}\Delta z_i{{}^2}_{{}^{}}}}{\mathrm{n}}}\\ {\delta }_{xy}=\pm \sqrt{{\delta }_x{}^2+{\delta }_y{}^2},{\delta }_p=\pm \sqrt{{\delta }_x{}^2+{\delta }_y{}^2+{\delta }_z{}^2}\end{array}$

where $\delta$_x is the median error of the points in the x-dimension, $\delta$_y is the median error of the points in the y-dimension, $\delta$_z is the median error of the points in the z-dimension, $\delta$_xy is the median error value of the points in the plane, and $\delta$_p is the 3D point error value.

This experimental study was aimed at the detection of the repeatable observation accuracy of the UAV. Both window sampling and DoD-DSM were used. Based on the accuracy analysis of the window sampling 3D point set, first, the DSM data constructed from the MVS dense reconstruction of the survey area were used as the basis, combined with the field survey and DOM image information, to distinguish between hardened and non-hardened ground surfaces within the survey area. Second, three 1 * 1 m window surfaces were selected for the 3D point set sampling in each of the two typical land classes, and the mean values of plane coordinates and mean values of elevation coordinates of the sampled point sets within each window in different sorties of DSM were calculated for statistical processing, and their point median error values were calculated using the above formulae in the same way [28]. DEM differential processing (DoD) based on different time series is currently one of the commonly used technical methods for terrain change and catastrophe detection. However, an analysis of the differences between DSMS obtained from repeated observations of the same, unchanging area can help to measure the limits of UAV measurement techniques for change detection [29].

In this study, the DSM, rather than the DEM, was chosen as the base data for the comparison of 3D measurement results between different repeat sorties of the UAV, in order to avoid error transfer during the processing of the DEM data, on the one hand, and to verify the agreement between the data through the DSM differential processing, on the other, which more reliably maps the photographic errors from the UAV technology system itself.

(2)$\begin{array}{l}DoD-DS{M}_1=H40J1-H40J2\hfill \\ DoD-DS{M}_2=H40J1-H40J3\hfill \\ DoD-DS{M}_3=H40J3-H40J2\hfill \end{array}$

As shown in Equation (2), the difference between the DSM data of different sorties was calculated, and a difference model was constructed. Where $DoD-DS{M}_1$ is the difference between sortie 1 and sortie 2. $DoD-DS{M}_2$ is the difference between sortie 1 and sortie 3. $DoD-DS{M}_3$ is the difference between sortie 3 and sortie 2.

3. Analysis of Test Results

3.1. Visualisation and Interpretation of Typical Land Classes

The DSM was acquired based on UAV imaging processing and visualized through different 3D scenes, as shown in Figure 3.

Figure 3a shows the DSM+ elevation rendering, which gives a general variation of the terrain within the survey area, which can be analyzed to show that the elevation range of the survey area is between 1933 m and 1946 m, with a relative height difference of 13 m, and that the elevation rendering shows a large area of elevation change, as long as it is due to changes in vegetation height. Figure 3b for DSM + intensity selector, this rendering mode can clearly see the range of the measurement area for the class of surface reflection intensity, in a reflection intensity range value of 2555–59,047; by calculating the intensity of the average value of 28,056.642, it is not difficult to show that the measurement area reflection intensity difference is large, reflecting that the surface structure of the area varies greatly. Figure 3c is the ‘DSM+Texture’ mapping, which maps the RGB color information from the UAV remote sensing images onto the DSM data, enabling the DSM data to have, not only accurate location information, but also clear texture information.

It is clear from Figure 3c, that there is a distribution of non-hardened (I) and hardened surfaces (II) with different surface material structures, and that the non-hardened surface areas have different heights of vegetation distribution.

As can be clearly seen from the 3D models in Figure 3d,e, the area is relatively flat and the vegetation is well distributed, concentrated in the non-hardened surface beds, with vegetation of a certain height distributed around the beds and mainly in grassed areas inside the beds.

3.2. Comparative Analysis of the 3D Accuracy of Typical Ground Classes Based on Window Sampling

In order to further detect the 3D accuracy of UAV measurement in different terrain types, the window sampling method of traditional remote sensing interpretation was used for reference, in combination with the actual situation of the test area, and exponentially expanded surface sampling was carried out. However, other features were easily added, which affected the results of the later experiment. Comprehensive comparison shows that a sampling surface of 1 * 1 m is more suitable.

In the study, the non-hardened surface ‘I’ and the hardened surface ‘II’, which were circled in conjunction with the decoding of Figure 3, were sampled in 3D point sets with a 1 m *1 m window of DSM data in each of the two types of surface; the distribution of the sampling windows is shown in Figure 4.

The ‘S1’ window surface is a non-hardened surface 3D point set with surface cover and uniform vegetation distribution, while the ‘S2’ window surface is a hardened surface 3D point set with a smooth and flat surface.

The 3D point sets were extracted from the S1 and S1 sampling windows, respectively, and then the 3D point sets were averaged to produce the 3D point set deviation results, as listed in Table 2; then the 3D point set error was calculated using the accuracy measurement formula of the point error analysis.

According to the calculation in Table 2, the 1 * 1 m window sampling 3D point set of different surface types was adopted to calculate the point position error analysis. The minimum error of I-S1 non-rigid surface was 0.001 m in the X direction, 0.004 m in the Y direction, and 0.001 m in the Z direction. The minimum point error of II-S2 non-rigid surface is 0.001 m in X direction, 0.002 m in Y direction, and 0.002 m in Z direction. Based on the point error analysis of multidimensional data, the surface plane error of the I-S1 class and 3D point errors were 0.007 m and 0.011 m, respectively. The surface plane errors of the class II-S2 and 3D point errors were 0.004 m and 0.009 m, respectively. It was found that the accuracy of the vertical direction was obviously lower than that of the horizontal direction, and most of the plane accuracy and elevation accuracy reached millimeter level. Combined with the analysis of the sampling accuracy of surface structure types, the accuracy error of hardened surfaces was generally smaller than the accuracy error value of non-hardened surfaces. Combined with the analysis of the 3D model visualization of the survey area, the presence of low vegetation, and the roughness of the non-hardened ground surface, it was found that the roughness of the ground surface had an influence on the accuracy of the SFM-DSM data; the less ground cover, the higher the accuracy of the data, and vice versa.

3.3. Comparative Analysis of UAV-3D Accuracy Based on DoD-DSM

The DSM is the most original data processing by SFM-MVS algorithm, with the accuracy closest to the real landform and editable. In this study, DSM1, DSM2, and DSM3 results of data collection from three UAV sorties and 3D model scene construction were analyzed. In order to further detect the variation of multi-period data, DSM was used as the basic data for point cloud differential processing, and a differential model was constructed according to the DoD–DSM formula in Section 2.3.

The differential change detection analysis of the point cloud in the LiDAR360 software processing platform not only enables the visual interpretation of change features from 3D space, but also ensures accuracy. Combined with the difference model, the change detection results are shown in Figure 5. In the difference model change detection, the grid size was 0.5 * 0.5 m and the elevation threshold was 1 m.

As shown in Figure 5, when the DoD-DSM difference model change of detection was carried out, the height difference changes were mainly concentrated between 0 and 1 m, and the change trend was roughly the same. Through comprehensive comparison of the three images, it was found that in the DoD-DSM3 model figure, the divination of the grid area between 0–0.2 m accounts for a larger proportion than in the other two models. In addition, the change value of height difference in the non-rigid surface area is mainly concentrated between 0 and 0.05 m.

Among these, the area with a height difference greater than 0.5 m was mainly around the flower beds on the non-rigid surface, and trees of different heights were distributed in this area. Among them, the white area of the picture refers to the hole in DSM caused by the loss of image, caused by the occlusion of trees during the shooting process. In order to further quantitatively describe the DoD-DSM changes, statistical analysis was conducted on the transformed area and volume, as shown in Table 3.

According to the calculation in Table 3, first of all, by comparing the three difference models, it is not difficult to find that the standard deviation (Std.D) of the three groups of difference models was ±0.067 m.

Of these, in the DoD-DSM2 difference model, the minimum root mean square error (RMSE) was ±0.082 m, the minimum elevation mean (Average dz) was 0.071 m, the minimum increase area was 707 m², the maximum decrease area was 577 m², and the minimum overlap area was 3926.5 m².

Second, the mean value of each indicator of the three groups of difference models was calculated, and a comprehensive comparison was made between each difference model and the mean value, to analyze the deviation degree of each difference model, so as to deduce the data coincidence degree between each flight. In Figure 6, the deviation statistics of each individual indicator from the difference model are shown separately.

As shown in Figure 6a, for all three sets of DoD-DSM models, the standard deviation (Std. D) value was 0.067 m; Figure 6b shows the root mean square error values for the three sets of DoD-DSM models, where the DoD-DSM2 error value was the smallest, with a value of 0.082 m; Figure 6c shows the elevation change values for the three sets of DoD-DSM models, where the DoD-DSM2 value was the smallest, with a value of 0.071 m; Figure 6d shows the increased area values for the three sets of DoD-DSM models, where the smallest value of DoD-DSM2 was 707 m²; conversely, Figure 6e shows the decreased area values for the three sets of DoD-DSM models, where the largest value of DoD-DSM2 was 577 m²; Figure 6f shows the overlapping area values for the three sets of DoD-DSM models, where the smallest value of DoD-DSM₂ was 3926.5 m².

The combination of the indicators shows that the indicators of DoD-DSM3 have the smallest deviation, and it can be concluded that the data of H100J2, H100J3, and DSM are close to each other, and the data fit is good. Similarly, it can be concluded that the error between H100J1 and H100J3 is larger, the two deviations are larger, and the transformation is more obvious. In summary, the standard deviation and root mean square error values can reach the centimeter level when the DoD-DSM difference model change detection is constructed based on the UAV repeated-observation SfM-MVS. Moreover, the DoD-DSM difference model can be used to infer the degree of agreement between the sortie data.

Combining the above studies, it is not difficult to see that the error in the construction of 3D models of trees, shrubs, etc. will be greater than that of grass. This is because the influence of the natural environment, such as the sun and wind cannot be controlled in repeated observations, resulting in the movement of leaves and branches in the higher parts of the vegetation, changing each time the data is acquired; while changes in the sun position may also lead to changes in the area and position of the vegetation shadows. Moreover, the quality of model construction based on the SfM method depends on the texture/color of the reconstructed surface.

4. Conclusions

In this paper, DJI Phantom 4 RTK consumer UAV was used for data acquisition; UAV repeated measurement was carried out at 40 m altitude, and DOM data and DSM data of the test area were obtained using the SFM-MVS method. Combining the two data sets and based on the 3D point set sampling of the window plane, the 3D point errors of repeated observation were calculated according to the general error analysis method of topographic survey for hardened and non-hardened surface stability. In order to further analyze the changes of different sorties under the same observation conditions and in the same processing environment, a DoD model was constructed to detect changes in DSM data for the three sorties.

The following conclusions are drawn: (1) The results of the 3D point set sampling test based on window plane show that the plane error and vertical error of the data obtained by repeated observation with the UAV are mostly up to millimeter level. Among them, the plane error of the hardened surface is ±0.004 m, and the mean error of the 3D point is ±0.009 m. The mean plane error of non-rigid surface is ±0.007 m, and the mean 3D point error is ±0.011 m. It is not difficult to see that the overall precision of the rigid surface is better than that of the non-rigid surface. (2) Comparative analysis of the accuracy of DOD-DSM UAV-3D; through analysis of the deviation value of the evaluation index of each station, it is concluded that the H40J2 and H40J3 DSM data are close and the deviation is small; H40J1 and H40J3 have large errors and a large deviation degree. By integrating the three difference models, the standard deviation and root mean square error are 0.063 m and 0.083 m, respectively, both of which can reach the centimeter level. Through the DSM 3D accuracy analysis of the above two methods, the error accuracy based on a window surface sampling method can be detected at the centimetric level, and the precision of the stabilized hard surface is better than that of the non-hard surface, on the whole. The main reason for this is the different surface roughness. Error accuracy analysis, based on the DoD-DSM difference model, can detect the coincidence between different time series data, but the surface change occurring at 6.7 cm can be detected; when the surface change is less than 6.7 cm using a DJI Phantom 4 RTK consumer grade UAV for transformation detection, the bias value brought in by the systematic error of data acquisition and processing needs to be considered. On the other hand, the purpose of the DoD-DSM difference model is to make a difference for DSM that has not been filtered as a whole in the survey area. The source of error was mainly from the noise introduced by vegetation with different heights in the survey area in the observation process.

In the course of this experimental study, repeatable observations of UAV were carried out, mainly from the perspective of field data acquisition, and the SFM-MVS method was used to obtain DSM data from repeatable observations and to detect the measurement limits of the UAV technology, through the differences between DSM data. It has been concluded through empirical tests that the accuracy of DSM data obtained from repeatable UAV observations can reach the centimeter level in plane and vertical accuracy, and even with most at the millimeter level. However, there are still some problems and shortcomings in the research results, and a comprehensive analysis of the measurements and accuracy due to sensor and software benchmarks is lacking. First, in the experimental study, only Agisoft Metashape was used for data processing for preliminary comparisons, ignoring the impact of other software benchmarks in 3D modelling on the accuracy when using the SFM method. Second, a single sample space was chosen, with only two land types selected, and no other land types covered by different features were analyzed in terms of 3D modelling accuracy. Finally, only a sample space of 1 * 1 m was selected for the sample space sampling surface; thus, lacking verification of the accuracy of 3D modelling data at different scales, making the results not universally applicable.

This team study is an initial exploration of repeat measurement accuracy and, therefore, lacks the depth required to delve into sensor and software benchmarking, but it sets the stage for more research to come. For example: (1) The test error factors can be extended from single factor, single scale to multiple factors, multiple scales. For example, the accuracy verification of data processing results for different altitudes, slopes, angles, sampling surfaces of different sizes, and different software benchmarks; (2) The accuracy assessment of UAV derivatives is relatively new, basically relying on external geometric parameters to make an assessment; how to assess the accuracy from physical parameters is a key direction for future research, and can also better reflect the practicality of UAV observation technology, etc.

Author Contributions

S.G. (Sha Gao) performed the research and methodology, analyzed the data, and wrote the manuscript. S.G. (Shu Gan) designed the framework of the research, mastered the conceptualization. S.G. (Shu Gan) and X.Y. have given many suggestions for improving and modifying this paper. R.L. and W.L. participated in data collection and investigation. R.B. and L.H. contributed to data processing, visualization analysis. All authors have read and agreed to the published version of the manuscript.

Funding

This research received the National Natural Science Foundation of China (No. 41861054 and No. 41561083).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data are not publicly available as they involve the subsequent application of other studies.

Conflicts of Interest

The authors declared that they have no conflicts of interest in this work. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables

Figure 1. Location map of the study area.

Figure 2. Flow chart of technical method.

Figure 3. Visual analysis (a) DSM+ elevation rendering; (b) DSM+ intensity rendering; (c) DSM+ texture mapping + Plot outline; (d) 3D white mold; (e) 3D model.

Figure 4. Geomorphology classification and sampling surface selection.

Figure 5. Change detection model diagram of DOD-DSM: (a) DOD-DSM1; (b) DOD-DSM2; (c) DoD-DSM3.

Figure 6. Statistical graph of the degree of deviation of each evaluation index in DOD-DSM model (a) Std.D value statistics; (b) RMSE value statistics; (c) Average value statistics; (d) Increased area value statistics; (e) Reduced area value statistics; (f) Overlap area value statistics.

Table 1

DJI Phantom 4 RTK drone platform and camera parameters.

UAV Platform		Camera Parameters
Model	DJI Phantom 4 RTK	Model	FC6310R
Fuselage weight (with paddles and battery)	1391 g	Picture size	5472 * 3648/pixel
Flight time	30 min	Sensor size	13.2 * 8.8 mm
Maximum take-off altitude	6000 m	Pixel dimension	2.4 μm
Maximum horizontal flight rate	50 km/h (Positioning mode)58 km/h (attitude mode)	Lens focal length	8.8 mm
Positioning systems	RTK GNSS	Field of view (FOV)	84°

Table 2

1 * 1 m window sampling 3D point set accuracy analysis table.

Land Type		Errors in Each Direction (m)
		$$\overline{\mathbf{X}}$$	$$\overline{\mathbf{Y}}$$	$$\overline{\mathbf{Z}}$$	$$\mathit{\delta }$$ x	$$\mathit{\delta }$$ y	$$\mathit{\delta }$$ z	$$\mathit{\delta }$$ xy	$$\mathit{\delta }$$ p
I-S1	DSM1	5···9.886	27···6.814	1934.933	0.004	0.009	−0.012	0.010	0.015
	DSM2	5···9.876	27···6.801	1934.956	−0.006	−0.004	0.011	0.007	0.013
	DSM3	5···9.883	27···6.801	1934.946	0.001	−0.004	0.001	0.005	0.005
Mean		5···9.882	27···6.805	1934.945	0.004	0.006	0.008	0.007	0.011
II-S2	DSM1	5···3.730	27···9.807	1934.888	0.003	0.003	−0.006	0.004	0.007
	DSM2	5···3.728	27···9.806	1934.905	0.001	0.002	0.011	0.002	0.011
	DSM3	5···3.723	27···9.800	1934.889	−0.004	−0.004	−0.005	0.006	0.008
Mean		5···3.727	27···9.804	1934.894	0.003	0.003	0.007	0.004	0.009

Note: when the mean is solved, the sum of absolute values is averaged. $$\overline{\mathrm{X}}$$, $$\overline{\mathrm{Y}}$$ and $$\overline{\mathrm{Z}}$$ are the mean values for each direction obtained by averaging the set of 3D points sampled from the corresponding window.

Table 3

Statistical table of the DoD model transformation detection.

DoD Model	Std.D	RMSE	Averagedz (m)	Increased Area (m2)	Reduced Area (m2)	Overlap Area (m2)
DoD-DSM1	0.067	0.084	0.076	753	554	3933.0
DoD-DSM2	0.067	0.082	0.071	707	577	3926.5
DoD-DSM3	0.067	0.084	0.075	711	567	3933.3
Mean	0.067	0.083	0.074	714	570	3930.9

References

1. Ciotină, M.C.; Niculiță, M.; Stoilov-Linu, V. LiDAR, UAV SFM and Geomorphic Change Detection in Small Quarry and Landslide Interactions. Proceedings of the EGU General Assembly 2021; Online, 19–30 April 2021; Conference Abstracts pp. EGU21-14651.

2. Mauri, L.; Straffelini, E.; Cucchiaro, S.; Tarolli, P. UAV-SFM 4D mapping of landslides activated in a steep terraced agricultural area. J. Agric. Eng.; 2021; 52, [DOI: https://dx.doi.org/10.4081/jae.2021.1130]

3. Anders, N.; Smith, M.; Suomalainen, J.; Cammeraat, E.; Valente, J.; Keesstra, S. Impact of flight altitude and cover orientation on Digital Surface Model (DSM) accuracy for flood damage assessment in Murcia (Spain) using a fixed-wing UAV. Earth Sci. Inform.; 2020; 13, pp. 391-404. [DOI: https://dx.doi.org/10.1007/s12145-019-00427-7]

4. Carbonneau, P.E.; Dietrich, J.T. Cost-effective non-metric photogrammetry from consumer-grades UAS: Implications for direct georeferencing of structure from motion photogrammetry. Earth Surf. Process. Landf.; 2017; 42, pp. 473-486. [DOI: https://dx.doi.org/10.1002/esp.4012]

5. Harwin, S.; Lucieer, A.; Osborn, J. The impact of the calibration method on the accuracy of point clouds derived using unmanned aerial vehicle multi-view stereopsis. Remote Sens.; 2015; 7, pp. 11933-11953. [DOI: https://dx.doi.org/10.3390/rs70911933]

6. Sanz-Ablanedo, E.; Chandler, J.H.; Rodríguez-Pérez, J.R.; Ordóñez, C. Accuracy of unmanned aerial vehicle (UAV) and SFM photogrammetry survey as a function of the number and location of ground control points used. Remote Sens.; 2018; 10, 1606. [DOI: https://dx.doi.org/10.3390/rs10101606]

7. Gindraux, S.; Boesch, R.; Farinotti, D. Accuracy assessment of digital surface models from unmanned aerial vehicles’ imagery on glaciers. Remote Sens.; 2017; 9, 186. [DOI: https://dx.doi.org/10.3390/rs9020186]

8. Shahbazi, M.; Sohn, G.; Théau, J.; Menard, P. Development and evaluation of a UAV-photogrammetry system for precise 3D environmental modeling. Sensors; 2015; 15, pp. 27493-27524. [DOI: https://dx.doi.org/10.3390/s151127493]

9. Lu, C.H. Applying UAV and photogrammetry to monitor the morphological changes along the beach in Penghu islands. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.; 2016; 41, pp. 1153-1156. [DOI: https://dx.doi.org/10.5194/isprs-archives-XLI-B8-1153-2016]

10. Siebert, S.; Teizer, J. Mobile 3D mapping for surveying earthwork projects using an Unmanned Aerial Vehicle (UAV) system. Autom. Constr.; 2014; 41, pp. 1-14. [DOI: https://dx.doi.org/10.1016/j.autcon.2014.01.004]

11. Bi, H.; Zheng, W.; Ren, Z.; Zeng, J.; Yu, J. Using an unmanned aerial vehicle for topography mapping of the fault zone based on structure from motion photogrammetry. Int. J. Remote Sens.; 2017; 38, pp. 2495-2510. [DOI: https://dx.doi.org/10.1080/01431161.2016.1249308]

12. Kong, J.; Gu, T.; Sun, P.; Zhu, L.; Sun, B.; Song, Z. Research on deformation evolution of landslides in Heifangtai silicified plantbased on multi-stage UAV images. J. Arid. Land Resour. Environ.; 2021; 35, pp. 40-107.

13. Zhou, T.; Hu, Z.; Han, J.; Zhang, H. Green vegetation extraction based on visible light image of UAV. China Environ. Sci.; 2021; 41, pp. 2380-2390.

14. Taddia, Y.; Stecchi, F.; Pellegrinelli, A. Coastal mapping using DJI Phantom 4 RTK in post-processing kinematic mode. Drones; 2020; 4, 9. [DOI: https://dx.doi.org/10.3390/drones4020009]

15. Wu, Y.; Yu, J.; Chen, R.; Yan, B. Research development of unmanned aerial vehicle-based oblique photogrammetry and its engineering applications. J. Hunan Univ.; 2018; 45, pp. 167-172.

16. Özcan, O.; Özcan, O. Multi-temporal UAV based repeat monitoring of rivers sensitive to flood. J. Maps; 2020; pp. 1-8. [DOI: https://dx.doi.org/10.1080/17445647.2020.1820387]

17. Daakir, M.; Pierrot-Deseilligny, M.; Bosser, P.; Pichard, F.; Thom, C.; Rabot, Y. Study of lever-arm effect using embedded photogrammetry and on-board GPS receiver on uav for metrological mapping purpose and proposal of a free ground measurements calibration procedure. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.; 2016; 9. [DOI: https://dx.doi.org/10.5194/isprsarchives-XL-3-W4-65-2016]

18. Forlani, G.; Dall’Asta, E.; Diotri, F.; Cella, U.M.D.; Roncella, R.; Santise, M. Quality assessment of DSMs produced from UAV flights georeferenced with on-board RTK positioning. Remote Sens.; 2018; 10, 311. [DOI: https://dx.doi.org/10.3390/rs10020311]

19. Grottoli, E.; Biausque, M.; Rogers, D.; Jackson, D.W.; Cooper, J.A.G. Structure-from-motion-derived digital surface models from historical aerial photographs: A new 3D application for coastal dune monitoring. Remote Sens.; 2021; 13, 95. [DOI: https://dx.doi.org/10.3390/rs13010095]

20. Nikolov, I.A.; Madsen, C.B. Performance Characterization of Absolute Scale Computation for 3D Structure from Motion Reconstruction. Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (Visigrapp 2019); Prague, Czech Republic, 25–27 February 2019; pp. 884-891.

21. Nikolov, I.; Madsen, C.B. Calculating absolute scale and scale uncertainty for SfM using distance sensor measurements: A lightweight and flexible approach. Recent Advances in 3D Imaging, Modeling, and Reconstruction; IGI Global: Hershey, PA, USA, 2020; pp. 168-192. [DOI: https://dx.doi.org/10.4018/978-1-5225-5294-9.ch008]

22. Li, W.; Yan, Q.; Zhang, S.; Wang, L.; Liu, D. Geomorphological data acquisition accuracy based on UAV and SFM. J. Basic Sci. Eng.; 2019; 27, pp. 1225-1234.

23. Lastilla, L.; Belloni, V.; Ravanelli, R.; Crespi, M. DSM Generation from single and cross-sensor multi-view satellite images using the new Agisoft Metashape: The case studies of Trento and Matera (Italy). Remote Sens.; 2021; 13, 593. [DOI: https://dx.doi.org/10.3390/rs13040593]

24. Peppa, M.V.; Hall, J.; Goodyear, J.; Mills, J.P. Photogrammetric assessment and Comparison of DJI Phantom 4 Pro and Phantom 4 RTK Small Unmanned Aircraft Systems. Proceedings of the International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS Geospatial Week 2019; Enschede, The Netherlands, 10–14 June 2019.

25. Alfio, V.S.; Costantino, D.; Pepe, M. Influence of image TIFF format and JPEG compression level in the accuracy of the 3D model and quality of the orthophoto in UAV photogrammetry. J. Imaging; 2020; 6, 30. [DOI: https://dx.doi.org/10.3390/jimaging6050030] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34460732]

26. Lim, P.C.; Kim, T.; Na, S.I.; Lee, K.-D.; Ahn, H.-Y.; Hong, J. Analysis of UAV image quality using edge analysis. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.; 2018; 62-64, pp. 359-364. [DOI: https://dx.doi.org/10.5194/isprs-archives-XLII-4-359-2018]

27. Zhang, C.; Yang, S.; Zhao, C.; Lou, H.; Zhang, Y.; Bai, J.; Zhang, Y. Topographic data accuracy verification of small consumer UAV. J. Remote Sens.; 2018; 22, pp. 185-195.

28. Xiu, C.; Li, Z.; Duan, H.; Lu, X.; Ma, T.; Shan, C. Plane Accuracy of Aerophotogrammetry without Image Control Points of DJI PHANTOM 4 RTK UAV. IOP Conference Series: Earth and Environmental Science; IOP Publishing: Bristol, UK, 2021; Volume 734, 012003.

29. Rock, G.; Ries, J.B.; Udelhoven, T. Sensitivity Analysis of UAV-Photogrammetry for Creating Digital Elevation models (DEM). Proceedings of the Conference on Unmanned Aerial Vehicle in Geomatics; Zurich, Switzerland, 14–16 September 2011; 1416.