The Tracking and Frequency Measurement of the Sway of Leafless Deciduous Trees by Adaptive Tracking Window Based on MOSSE

Abstract

The tree sway frequency is an important part of the dynamic properties of trees. In order to obtain trees sway frequency in wind, a method of tracking and measuring the sway frequency of leafless deciduous trees by adaptive tracking window based on MOSSE was proposed. Firstly, an adaptive tracking window is constructed for the observed target. Secondly, the tracking method based on Minimum Output Sum Of Squared Error Filter (MOSSE) is used to track tree sway. Thirdly, Fast Fourier transform was used to analyze the horizontal sway velocity of the target area on the trees, and the sway frequency was determined. Finally, comparing the power spectral densities (PSDs) of the x axis acceleration measured by the accelerometer and PSDs of the x axis velocity measured by the video, the fundamental sway frequency measured by the accelerometer is equal to the fundamental sway frequency measured by video. The results show that the video-based method can be used successfully for measuring the sway frequency of leafless deciduous trees.

1. Introduction

Wind has an important influence on a single tree or whole forest stands and is one of the main causes of forest disasters. Trunks and branches bend and spin to a certain degree under the action of wind. Under the action of strong winds, the wind-induced load of trees is too high, and the trees break or are uprooted [1,2], which reduces the timber yield of trees and causes economic loss [3,4,5]. The tree sway frequency is an important part of the dynamic properties of trees. The more the tree frequency is far from the wind frequency, the higher is the safety. The fundamental sway frequency at which a tree would tend to sway is an important research focus because sway magnitude could enlarge rapidly at this frequency and lead to possible failure [6,7]. Researchers have conducted a number of outdoor experiments to measure tree sway frequencies, which can be divided into two types depending on the source of the tree’s vibration excitation. The first is enacted during a time of little natural wind. A rope and winch is used to pull back the trees. When the trees is released, the free motion of the trees is recorded. This type of experiment is commonly called a “pull test” [8]. The second is the tracking of sway motion under natural wind conditions, which can truly reflect the dynamic properties of trees [6,7,9,10,11]. Therefore, a method that can measure tree sway in the wind is of great significance to understand the dynamic properties of trees.

In order to monitor the sway and frequency response of trees, a variety of different motion capture methods are used. Researchers have used an accelerometer to measure the trees sway [12,13,14]. Roodbaraky [15] measured the trees sway using potentiometers attached to the trees. Nikolas [10] used a strain gauge to measure the trees sway. In addition, researchers have used tilt sensors to measure the trees sway [6,11,16]. The advantage of these systems is that they can monitor and record signals for months. However, the above methods all involve the installation of instrumentation directly on the target part of a tree; they may require a power supply, and the instruments themselves may impact the sway of the tree part with the addition of their mass. In addition, when measuring small objects such as branches, the added mass of accelerometers may affect the measurement results.

A variety of non-contact equipment has also been used to capture trees sway. Baker [17] used a laser Doppler interferometer to measure the trees sway. The device measures the power spectrum of trees velocity and obtain frequencies of trees. The laser method is expensive and less practical. Hassinen [18] used the prism system to measure the trees sway in the wind. The prism-based system consists of an infrared emitting diode, infrared transmitter, optical receiver, and controller unit, which are placed in a cylinder at the base of the trees. Barbacci [19] used magnetic sensors to measure the displacement of the trees. While magnetic sensors can be used to capture the displacement of trees branches swaying in the wind, they can only measure at a limited distance, and are complex to install and expensive. Hu [20] used a Microsoft Kinect V2, which can capture RGB and depth images, to capture the movement of trees and measure the trees sway. Kinect method can capture the three-dimensional trajectory of trees, but the Kinect method is greatly affected by light. In addition, most non-contact measurements are not conducive to the capture of tall trees, especially high aerial branches. For example, magnetic sensors and Kinect have limited capture distance to target movement, while laser and prism methods have limited measurement distance due to occlusion.

With the development of computer vision technology, some researchers have used video methods to capture the sway of trees in recent years. The pyramidal implementation of the Kanade-Lucas-Tomasi (KLT) algorithm [21] by Bouguet [22] for feature tracking is the most commonly used method to capture trees sway based on video method. Diener [23] used the KLT algorithm to track the sway of miniature and potted tree. Barbacci [19] used the KLT method to track a 10-year-old walnut potted tree sway and measured the sway displacement of trees in the wind. Der Loughian [24] used the KLT algorithm to capture the movement of small bonsai trees and measure their vibration characteristics. KLT algorithm is a kind of optical flow method, the classical method in computer vision. In this method, the cause of the local changes in gray level intensity is assumed to be relative motion between the camera and the trees. The material derivative of the change in gray level is thus zero and at some point in the image, the local derivative of the gray level intensity is equal to the motion velocity times the spatial gradient of gray level intensity, allowing the motion to be estimated. The KLT tracker might suffer an aperture problem, which is an optical illusion making the local determination of the movement impossible. The corner points (“good tracking feature”) [25] created by the method implemented by Bouguet [22] solves this problem. However, as the corner points are generated by the algorithm, it is impossible to guarantee that the feature points are on the branches of interest. When trees sway, this feature is likely to disappear in some frames, which leads to tracking failure. In addition, if the tracking is carried out for a long time (10–30 min or longer), it is very likely that the tracking will be lost due to the illumination change. Xue [26] used the dense optical flow method to track the sway of individual branches of the aerial tree part. In this method, all nodes of the branches in the first frame should be manually labelled, and then tracked by calculating the dense motion field. This method not only needs to be labelled the first frame, but also takes a long time to compute the dense field simultaneously. In previous studies, the data of 10 or 30 min were used to calculate the trees sway frequency. This method is too slow to handle such a long period of video. In addition, Nakata [27] obtained sway frequency by measuring the x-direction displacement of Arabidopsis through video. De Langre [28] measured the frequency of indoor poplar young plant seedlings using a high-speed camera. Therefore, the main problem of the previous video-based trees sway tracking algorithm is that it cannot guarantee that the tracked points are on the branches of interest. In addition, if the tracking is carried out for a long time, it is very likely that the tracking will be lost due to the change of illumination. If dense optical flow tracking is carried out, it requires a lot of computation and takes a long time to process, so it is not suitable for tracking trees sway for a long period of video data. To address the above problems, a method is needed to measure trees sway in nature that is insensitive to light conditions, can track the selected feature consistently, and has a processing speed that is suitable for long periods of data collection.

The target tracking method based on the Minimum Output Sum Of Squared Error Filter (MOSSE) is robust to illumination, size, attitude, and non-rigid deformation of a target [29]. Trees rotate and deform under the force of wind, but this algorithm effectively tracks a target as it undergoes such movement. In addition, its computation speed is the fastest among current tracking algorithms [30]. This method does not require preprocessing of video data and tracking areas can be selected as required. However, the tracking effect of this method is affected by the size of the tracking window. As the tracking window is small, the initial training sample cannot get enough features when the affine changes, so the phenomenon of tracking loss will appear in the process of tracking trees sway. Although a large tracking window will reduce drift, it will increase the amount of computation. At the same time, if multiple branches are covered, the final measured frequency will be affected by multiple branches and be inaccurate.

For these reasons, a method of trees sway tracking and frequency measurement by adaptive tracking window based on MOSSE is proposed. This method can make the tracking window size reasonable to ensure the tracking effect and reduce the amount of computation. In this study, the sway of two features of one tree are tracked via the MOSSE method, and sway frequencies were obtained using Fast Fourier transform (FFT) analysis of the horizontal sway velocities of these features. The results were found to be comparable to frequencies derived from sway data obtained with an accelerometer. The purpose of this study is to obtain the sway frequency of trees in wind by a video-based method.

2. Materials and Methods

2.1. Data Acquisition and Test System

A naturally growing perennial Betula platyphylla Sukaczev tree with a height of 23.2 m and diameter at breast height (DBH) of 27.4 cm was selected as the experimental object. The total station and trigonometric elevation method are used to measure the height of the tree. DBH is the tree diameter measured 1.3 m above the ground. It was located in the Maoershan Forest Ecosystem National Field Scientific Observation and Research Station (127°39′50″ E, 45°24′32″ N) in Shangzhi City, Heilongjiang Province. The camera (HSAB-IPC3219H, Huabang, ShenZhen, China) used for video recording was fixed at 4.5 m high on a steel pillar about 15.9 m from the bottom of the tree. The camera angle respect to the horizontal line was about 25°. The resolution of the camera was 1920 × 1080, and the focal length was set at 35 mm. A three-axis accelerometer (AIT2500, Jingming, Yangzhou, China) was set at 7.9 m up from the soil, below the lower tree knot of the tree to directly measure the trunk sway; the sampling frequency of the accelerometer was 50 Hz. This section is designated as S_1,a. The wind speed was measured by a three-axis ultrasonic anemometer (WindMaster, GILL, Lymington, Britain) installed at 20 m of the anemometer tower. The distance between the anemometer tower and the experimental object was 10 m. The measurement range of the ultrasonic anemometer is 0−45 m·s⁻¹, the resolution is 0.01 m·s⁻¹, and the output frequency was set to 10 Hz. The schematic diagram of the test system is shown in Figure 1.

In order to avoid the added complexity of leaves when targeting branches and trunks for measurement, video monitoring was done in autumn, during periods of strong wind (7–8 November 2019). Monitoring samples were 30 min in length. A total of 10 30-min sample videos were taken. The average wind speed during the monitoring period was 3.33 m·s⁻¹, and the maximum wind speed reached 12.28 m·s⁻¹.

Two positions on the tree in the video were selected for tracking and measurement. The area of position 1 covering a segment of the major primary branch is designated as S_1,v. The area of position 2 covering a segment of a branch on the major primary branch is designated as S_2,v. The positions of the monitoring areas are shown in Figure 2.

The accelerometer can be used to measure the frequency at tree trunk. In this study, it is used to validate the frequency measured by video. The sampling frequency of the video is 16 Hz. Based on the formula of Bunce [6], the fundamental sway frequency of the tree is estimated to be 0.26 Hz according to its height and DBH, when height to the base of the live crown (GRTOCR) is ignored in the condition of leaf-off-above-freeze. Due to the limited resolution of the video, small amplitude movement may not be detected. In order to have a better tracking effect and improve the computing efficiency, the sampling frequency of 4Hz, which is enough to meet the measurement requirements of low-order sway frequency of trees according to the Nyquist-Shannon sampling theorem that the sampling frequency should be greater than 2 times of the highest signal frequency, was selected to measure the sway frequency of the trees.

2.2. Method for Tracking and Frequency Measurement of Tree Sway

2.2.1. Minimum Output Sum of Squared Error Filter (MOSSE)

The minimum output sum of square of error filter (MOSSE) is a kind of algorithm which generates tracking data using a small number of training images [29]. MOSSE is used to track tree sway.

With the given training sample ${\mathrm{f}}_{\mathrm{i}}$ and expected output ${\mathrm{g}}_{\mathrm{i}}$, the corresponding filter ${\mathrm{h}}_{\mathrm{i}}$ is calculated. For the desired output ${\mathrm{g}}_{\mathrm{i}}$, the MOSSE filter is defined by a two-dimensional Gaussian distribution. The peak value is obtained at the center of the training image ${\mathrm{f}}_{\mathrm{i}}$, and the output value decreases with distance from the centre. The image is adjusted so that the target feature is in the centre position ${(\mathrm{x}}_{\mathrm{i}}{,\mathrm{y}}_{\mathrm{i}})$, and the expected output of the training image is defined as (1):

(1)${\mathrm{g}}_{\mathrm{i}}{(\mathrm{x},\mathrm{y})=\mathrm{e}}^{-\frac{{(\mathrm{x}-{\mathrm{x}}_{\mathrm{i}})}^2{+(\mathrm{y}-\mathrm{y}}_{\mathrm{i}}{)}^2}{{\mathsf{\sigma }}^2}},$

$\mathsf{\sigma }$ is the variance of the two-dimensional Gaussian distribution. In order to reduce the computational complexity in the process of filter training, FFT can be used to transform the input image ${\mathrm{f}}_{\mathrm{i}}$ and the expected output ${\mathrm{g}}_{\mathrm{i}}$ into the frequency domain, and simple inter-element multiplication can be used to solve for the filter. After FFT, input image ${\mathrm{f}}_{\mathrm{i}}$, expected output ${\mathrm{g}}_{\mathrm{i}}$ and filter $\mathrm{h}$ are respectively represented as ${\mathrm{F}}_{\mathrm{i}}$, ${\mathrm{G}}_{\mathrm{i}}$ and $\mathrm{H}$. Thus, the filter can be obtained by (2):

(2)${\mathrm{H}}_{\mathrm{i}}^{*}=\frac{{\mathrm{G}}_{\mathrm{i}}}{{\mathrm{F}}_{\mathrm{i}}},$

The division calculation is the division between elements. In order to minimize the error between the output of the convolution of the filter and the sample image and the set expected value ${\mathrm{G}}_{\mathrm{i}}$, the quadratic error minimization criterion is used for optimization, and the formula is shown in (3):

(3)$\underset{{\mathrm{H}}^{*}}{\min }{\displaystyle \sum }_{\mathrm{i}}{\left|{\mathrm{F}}_{\mathrm{i}}\odot {\mathrm{H}}^{*}-{\mathrm{G}}_{\mathrm{i}}\right|}^2,$

where $\odot$ represents the dot multiplication of the corresponding position. In the above optimization problem, the optimization function is a real valued function containing complex variables. ${\mathrm{H}}^{*}$ and ${\mathrm{F}}_{\mathrm{i}}^{*}$ are used to represent the conjugate variables of $\mathrm{H}$ and ${\mathrm{F}}_{\mathrm{i}}$ respectively. After solving, the approximate solution of ${\mathrm{H}}^{*}$ is obtained, as shown in (4):

(4)${\mathrm{H}}^{*}=\frac{{{\displaystyle \sum }}_{\mathrm{i}}{\mathrm{G}}_{\mathrm{i}}\odot {\mathrm{F}}_{\mathrm{i}}^{*}}{{{\displaystyle \sum }}_{\mathrm{i}}{\mathrm{F}}_{\mathrm{i}}\odot {\mathrm{F}}_{\mathrm{i}}^{*}},$

$\mathrm{H}$ is the formula in the MOSSE filter.

2.2.2. Tracking Algorithm Based on MOSSE

Using the above method to calculate the filter variable $\mathrm{H}$, the MOSSE algorithm can analyse video and create a data set of the sway velocities of targeted tree features over a period of time. The process is as follows:

(1) Initialization: During the training process of the filter, multiple images are obtained through random affine transformation of the initial target window in the first frame of the video. In this process, the image rotates and the target displacement is inaccurate. To solve this problem, the gray value of the image is normalized to an average of zero, and then the image is multiplied by a cosine window to make the edge value of the image close to zero, which highlights the weight of the intermediate target more. After the normalization and cosine processing, the training sample of the target was obtained, and the corresponding two-dimensional Gaussian distribution was defined as ${\mathrm{g}}_{\mathrm{i}}$.

(2) Establish the target appearance model: ${\mathrm{f}}_{\mathrm{i}}$ and ${\mathrm{g}}_{\mathrm{i}}$ are converted into frequency domain variables ${\mathrm{F}}_{\mathrm{i}}$ and ${\mathrm{G}}_{\mathrm{i}}$ by FFT, and $\mathrm{H}$ is obtained according to (4).

(3) Motion estimation: $\mathrm{H}$ is used to filter the search threshold of new frame $\mathrm{T}$ to obtain the response ${\mathrm{G}}_{\mathrm{T}}$, and the inverse Fourier transform (IFFT) is applied to it to obtain the spatial domain response ${\mathrm{g}}_{\mathrm{T}}$, where ${\mathrm{g}}_{\mathrm{T}}$ contains the target position information.

(4) Target positioning: the position of the peak value of the spatial domain response ${\mathrm{g}}_{\mathrm{T}}$ is the central position of the target.

(5) Occlusion and failure detection: The Peak to Sideline Ratio (PSR) is used to determine the target occlusion and measure the quality of tracking results. PSR is defined as (5):

(5)$\mathrm{PSR}=\frac{{\mathrm{g}}_{\max }{-\mathrm{u}}_{\mathrm{s}1}}{{\mathsf{\sigma }}_{\mathrm{s}1}},$

${\mathrm{g}}_{\max }$ is the peak, ${\mathrm{u}}_{\mathrm{s}1}$ and ${\mathsf{\sigma }}_{\mathrm{s}1}$ are the mean and standard deviation of the side lobes.

(6) Target appearance update: the tracking result of processed frame and current frame is considered comprehensively, and the learning rate is introduced to adapt to the change of target appearance and prevent drift. ${\mathrm{A}}_0$ and ${\mathrm{B}}_0$ are 0 initially, and $\mathrm{T}$ is the frame number. The update is as shown in (6)–(8):

(6)${\mathrm{H}}_{\mathrm{T}}^{*}=\frac{{\mathrm{A}}_{\mathrm{T}}}{{\mathrm{B}}_{\mathrm{T}}},$

(7)${\mathrm{A}}_{\mathrm{T}}{=\mathsf{\eta }\mathrm{G}}_{\mathrm{T}}\odot {\mathrm{F}}_{\mathrm{T}}^{*}{+(1-\mathsf{\eta })\mathrm{A}}_{\mathrm{T}-1},$

(8)${\mathrm{B}}_{\mathrm{T}}{=\mathsf{\eta }\mathrm{F}}_{\mathrm{T}}\odot {\mathrm{F}}_{\mathrm{T}}^{*}{+(1-\mathsf{\eta })\mathrm{B}}_{\mathrm{T}-1},$

$\mathsf{\eta }$ equals zero means that the filter is never updated. When $\mathsf{\eta }$ is large enough, the updated filter is most affected by the current frame, and the influence of the tracked frame on the filter decreases exponentially with time.

(1)–(4) were composed from the view point of linear evaluation. (6)–(8) divide the model formula of the filter into two parts, molecular and denominator. Each part is updated separately to be more robust to external influences such as deformation and illumination. In most cases, the local motion of the trunk and branches in wind is linear rotation or slightly distorted, so it is not necessary to consider the nonlinearity.

2.2.3. Adaptive Construction of Trace Window

To track tree sway with the MOSSE algorithm, the adaptive construction of a tracking window must be completed first. Tree sway is a recurrent movement, so an effective tracking window can be defined using only enough frames to depict a single complete sway motion of the targeted feature.

When the tracking position is selected, a tracking window that just covers the branch to be tracked is defined. For a video of duration $\mathrm{T}$, the first $\mathrm{T}/\mathrm{n}\left(\mathrm{n}\ge 5,\mathrm{n}\mathrm{indicates}\mathrm{that}\mathrm{the}\mathrm{video}\mathrm{is}\mathrm{divided}\mathrm{into}\mathrm{n}\mathrm{segments}\right)$ segments of the video track every m frames. If all trace is successful, consider the trace window size appropriate, and return. Otherwise, increase the tracking window according to the step size $\mathrm{s}\left(\mathrm{s}\ge 1\right)$ and re-track. The algorithm flow chart is shown in Figure 3.

2.2.4. Method for Measuring Tree Sway Frequency

Trees kept swaying in the wind, which makes it difficult to determine the initial position of trunks and branches in the windless state. In strong winds, the movement of the branch includes its own movement and movement of the trunk, so it is difficult to measure the absolute displacement of the trunk and branch, and it is difficult to measure the sway frequency of any targeted feature via its own movements, while it is easier to measure the relative displacement of the trunk and branch. The instantaneous velocity of that relative displacement, measured with the MOSSE tracking algorithm, can be used to calculate tree sway frequency.

Trees sway consists of motion in three dimensions. Z is the line-of-site of the camera, x and y are the horizontal and vertical axes perpendicular to line-of-site of the camera. Frequency is a directionless characteristic, and so the direction or dimension chosen for measuring movement is irrelevant. A video display is 2-dimensional, and movement along the x axis and y axis of that 2-dimensional plane are most easily discriminated. In the process described in this study, movement along the x axis is measured, and the sway frequency response can be obtained by FFT of the instantaneous average velocity along the x axis. Select the upper left corner of the rectangular tracking window as the feature point, and obtain the coordinates of the feature points of the tracking window in the process of tracking the movement of trees. The change of coordinate of feature points in this process is used to calculate the average velocity along the x axis between two frames of video, as shown in (9):

(9)${\mathrm{v}}_{\mathrm{x}}=\frac{{\mathrm{X}}_{\mathrm{i}}{-\mathrm{X}}_{\mathrm{i}-1}}{\mathsf{\Delta }\mathrm{t}},$

${\mathrm{v}}_{\mathrm{x}}$ is average velocity which can be considered as the instantaneous velocity between two frames of video, ${\mathrm{X}}_{\mathrm{i}}$ is the coordinate value of the feature point along the x axis on frame $\mathrm{i}$, ${\mathrm{X}}_{\mathrm{i}-1}$ is the coordinate value of the feature point in the x direction on frame $\mathrm{i}-1$, and $\mathsf{\Delta }\mathrm{t}$ is the time interval between two adjacent frames. The FFT is performed on the velocities calculated through the video set. The velocities through the video form the data set on which the FFT is performed. The FFT produces spectral information which can be displayed in the form of a spectrum. The frequency at which the greatest energy is found represents the spectral peak, which is identified as the frequency of the targeted feature.

2.2.5. Method Flow

The method of tree sway tracking and frequency measurement by adaptive tracking window based on MOSSE includes the following steps. First, an adaptive tracking window is constructed for the observed target. Second, the MOSSE algorithm is used to track selected features as they move within the frame. Third, the frequency of the tree is obtained by FFT analysis of the sway velocity of trees along the x-axis and the peak frequency is identified.

3. Results

3.1. Wind Speed

Using an ultrasonic anemometer, we can obtain the vector data of wind speed in three directions, namely $({\mathrm{U}}_{\mathrm{x}},{\mathrm{U}}_{\mathrm{y}},{\mathrm{U}}_{\mathrm{z}})$ in the cartesian coordinate system, and calculate the absolute value of wind speed $\mathrm{U}=\sqrt{{\mathrm{U}}_{\mathrm{x}}{}^2+{\mathrm{U}}_{\mathrm{y}}{}^2+{\mathrm{U}}_{\mathrm{z}}{}^2}$. The time history curves of absolute values of wind speed in the first four monitoring periods are shown in Figure 4.

3.2. Coordinate Change of Target Feature Points along the x Axis in Videos

The coordinate changes of target features of S_1,v, S_2,v along the x axis in the videos are obtained by using the proposed MOSSE method, and are displayed in Figure 5. The abscissa is the time, and the ordinate is the coordinate value of the feature points along the x axis in the pixel coordinate system. In the pixel coordinate system, the vertex in the upper left corner of the image plane is taken as the origin, and the x axis and y axis are parallel to the x axis and y axis of the image coordinate system, respectively, in pixels. The number of pixels represents the size of the target feature point moving along the x axis; it is proportional to the distance the target moves in the physical world in the same direction without changing the focal length.

It can be seen from Figure 5 that the coordinate values of the feature points along the x axis in the first frame in different monitoring periods are different, which indicates that the initial positions of S_1,v and S_2,v in the video are different. The abscissa of S_1,v and S_2,v fluctuated over time, indicating that the tree began to sway with the wind. Each extreme point in the graph indicates that the tree reaches its maximum movement in one direction and then starts moving in the opposite direction. The sharp part of the graph indicates that the tree sway speed is fast, while the gentle part indicates that the tree sway speed is slow. S_1,v and S_2,v have trajectories of similar shape, S_2,v showed larger fluctuations in coordinates than S_1,v.

3.3. The Velocity of the Feature Point along the x Axis in the Videos

Equation (9) was used to calculate the velocity time history curves of S_1,v and S_2,v. The velocity time history curves of feature points of S_1,v were shown in Figure 6. The abscissa is time and the ordinate is the velocity of the feature point along the x-axis.

It can be seen from Figure 6 that the initial speeds of the first frame are different in different monitoring periods. As time changes, the speed begins to fluctuate. If the value of the speed is not zero, it indicates that the tree is swaying. If the value of the speed is zero, it indicates that the tree is at rest, or represents the extreme point in one direction; at this time, it will start to move in another direction.

3.4. Comparison of Frequency Measured by a Video and an Accelerometer

Spectral analysis using FFT in OriginPro9.1 was used to calculate the frequencies of the tree. The power spectral density (PSD) of the x axis acceleration measured by the accelerometer and the x axis velocity measured by the video provides a measure of the tree’s response as a function of frequency. According to the sampling theorem, the maximum signal frequency obtained from video is 2 Hz. Therefore, for comparison, the frequency range of power spectrum obtained by an accelerometer and video is limited to 2 Hz. The power of each spectrum was averaged over a bandwidth of 0.01 Hz in order to enhance the statistical stability of the estimates [18]. The 10 groups of individual PSDs of acceleration of S_1,a and their average PSD are shown in Figure 7a. The 10 groups of individual PSDs of velocity of S_1,v and S_2,v and their average PSD are shown in Figure 7b,c. The PSD values of S_1,a, S_1,v, and S_2,v were adjusted to 0–1 using Normalize in OriginPro9.1 (OriginLab, Northampton, Massachusetts, USA) for comparison, as shown in Figure 7d.

As can be seen from Figure 7, the spectra resembled the Type I or III reported by Baker [17], with a recognizable peak between 0.2 and 0.4 Hz. The peak frequencies of the acceleration average PSD spectrum of S_1,a and the velocity average PSD spectrum of S_1,v are 0.27 Hz. The peak frequency of the velocity average PSD spectrum of S_2,v is 0.26Hz. The peak frequency is the fundamental sway frequency of the tree. The proposed method can be used for motion tracking during leafless periods and for fundamental vibration mode detection. In addition, the frequencies of S_1,a, S_1,v are equal, and the frequencies of S_2,v are close to S_1,a, and S_1,v. This is consistent with the results of Baker [17], where multiple points on a tree were measured and the frequencies of these points were close. This is because frequencies along the same tree are always the same at each point; what changes is vibration amplitude and acceleration due to variable damping along the tree structure.

4. Discussion

From a comparison with the measurement results of the accelerometer, it can be seen that the frequency of the target can be identified by FFT analysis of the motion signal of the target based on the video, which is comparable with the measurement results of the accelerometer. The method based on video can be used to collect data conveniently—a video can be recorded at any time with digital cameras or even mobile phones. In addition, although only the leafless period of deciduous species is monitored, there is no limit to the leafless period if the monitoring target is the trunk. In addition, monitoring of fully leaved trees is a future research direction.

Although video has the advantages of easy data collection and visualization, it still has its limitations. When collecting video data, it is easy to be affected by collection equipment and natural environment, such as camera shaking, light, rain, snow, and fog, and it is not possible to monitor trees in dense forests. Due to the influence of these factors, there will be a variety of noises and disturbances in the image, which will affect the monitoring effect of the video method on tree sway. The method does not work at night due to light. In rainy, snowy, and foggy weather, tracking will fail due to video blur. Target tracking in a hazy video is a typical problem. Chaudhary [31] studied this problem and proposed ResINet, which is a novel end-to-end convolution neural network for image de-hazing. We have not studied this question, which will be a more in-depth study. However, it should be noted that our method is mainly used to monitor trees in the leafless period, during which there is less rain. In addition, a single camera cannot capture the three-dimensional movement of trees; we only monitor the movement along the x axis in the video.

In the case of strong winds or high winds, parts of other trees will collide with parts of the sample tree. When the target is occluded by a small part or the background changes little, it has little impact on the tracking effect. The principle of MOSSE algorithm is to generate a high response (correlation peak) for each interested target in the scene and a low response for the background. The position corresponding to the maximum value in the expected output is the new position of the tracked target. Therefore, when the target is partially occluded or the background changes little, the tracking target will always correspond to the correlation peak. In addition, Peak to Sidelobe Ratio (PSR) is used to determine the quality of tracking results during the tracking process. If a large area of occlusion occurs, the PSR value will determine the tracking failure, and the tracking will continue when the target reappears.

In high winds, there may be greater deformation and sway of trunks and branches. The MOSSE algorithm is robust to the sway amplitude and small deformation of tracking target; the sway amplitude, especially, hardly affects the tracking effect. However, if the stiffness of the tree is low and large bending occurs, it may lead to tracking failure. If the stiffness is large enough, the tracking target only has small deformation, which will not lead to tracking failure.

5. Conclusions

An adaptive tracking window is constructed based on the MOSSE algorithm, and a method for tracking and measuring the frequency of leafless deciduous tree sway is established; the fundamental mode of the tree sway is analyzed. A Betula platyphylla Sukaczev tree located in the Maoershan Forest Ecosystem National Field Scientific Observation and Research Station was taken as the research object. The results measured by the video tracking method were compared with those measured by the accelerometer to verify the feasibility and effectiveness of the method. The main conclusions are as follows:

• The video-based method can be used successfully for measuring tree sway frequency under field conditions. The fundamental sway frequency measured by the accelerometer is equal to the fundamental sway frequency measured by the video.

• The key to this method is the construction of an adaptive tracking window. The two problems owing to which tracking fails—tracking window being too small and tracking speed and measurement accuracy are reduced due to the tracking window being too large—is addressed with the adaptive tracking window. The instantaneous velocity of the tree is calculated, and the frequency response of the tree is obtained by using FFT for spectrum analysis of instantaneous velocity.

• The frequency identification method of trees is based on the tracking method based on MOSSE, which ensures that the method is robust and fast and can track tree sway for a long time. In addition, the installation of the equipment is simple; thus, the method has cost efficiency performance in frequency measurement.

However, this method does not work at night, might not work for strong winds, and is only for leafless trees. In addition, a single camera cannot obtain tree depth information. In future, we will study the tracking and measurement of three-dimensional movement of trees and extend our research to multi-species and whole-leaf trees.

Author Contributions

D.X., A.W. and X.Y. designed the experiment. A.W. and X.Y. carried out the experiments, analysed the data, and wrote the original manuscript with the help and critique of D.X., A.W. and X.Y. contributed equally to this work. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Natural Science Foundation of Heilongjiang Province of China, grant number LH2020C091; the Natural Science Foundation of Heilongjiang Province of China, grant number LH2021F050.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Figure 1. Schematic diagram of the test system.

Figure 2. (a) The positions of S1,v, S2,v and anemometer tower in the camera image; (b) Positions of S1,a and S1,v.

Figure 2. (a) The positions of S1,v, S2,v and anemometer tower in the camera image; (b) Positions of S1,a and S1,v.

Figure 3. Flowchart of the adaptive construction method of the tracking window.

Figure 4. The time history curves of absolute values of wind speed in the first four monitoring periods.

Figure 5. The coordinate change of feature points of S1,v, S2,v in the x axis direction in the videos in the first four monitoring periods.

Figure 6. The velocity time history curve of the feature points of S1,v in the first four monitoring periods.

Figure 7. (a) The 10 groups of individual PSD of acceleration of S1,a and their average PSD; (b) The 10 groups of individual PSD of velocity of S1,v and their average PSD; (c) The 10 groups of individual PSD of velocity of S2,v and their average PSD; (d) Normalized PSD of S1,a, S1,v, S2,v.

References

1. Gardiner, B.; Blennow, K.; Carnus, J.-M.; Fleischer, P.; Ingemarson, F.; Landmann, G.; Lindner, M.; Marzano, M.; Nicoll, B.; Orazio, C. et al. Destructive Storms in European Forests: Past and Forthcoming Impacts. Available online: https://ec.europa.eu/environment/forests/pdf/STORMS%20Final_Report.pdf (accessed on 19 June 2019).

2. Schindler, D.; Mohr, M. Non-oscillatory response to wind loading dominates movement of Scots pine trees. Agric. For. Meteorol.; 2018; 250–251, pp. 209-216. [DOI: https://dx.doi.org/10.1016/j.agrformet.2017.12.258]

3. Bayat, M.; Ghorbanpour, M.; Zare, R.; Jaafari, A.; Pham, B.T. Application of Artificial Neural Networks for Predicting Tree Survival and Mortality in the Hyrcanian forest of Iran. Comput. Electron. Agric.; 2019; 164, 104929. [DOI: https://dx.doi.org/10.1016/j.compag.2019.104929]

4. Bayat, M.; Noi, P.T.; Zare, R.; Bui, D.T. A Semi-Empirical Approach Based on Genetic Programming for the Study of Biophysical Controls on Diameter-Growth of Fagus Orientalis in Northern Iran. Remote Sens.; 2019; 11, 1680. [DOI: https://dx.doi.org/10.3390/rs11141680]

5. Bayat, M.; Bettinger, P.; Heidari, S.; Hamidi, S.K.; Jaafari, A. A Combination of Biotic and Abiotic Factors and Diversity Determine Productivity in Natural Deciduous Forests. Forests; 2021; 12, 1450. [DOI: https://dx.doi.org/10.3390/f12111450]

6. Bunce, A.; Volin, J.C.; Miller, D.R.; Parent, J.; Rudnicki, M. Determinants of tree sway frequency in temperate deciduous forests of the Northeast United States. Agric. For. Meteorol.; 2019; 266–267, pp. 87-96. [DOI: https://dx.doi.org/10.1016/j.agrformet.2018.11.020]

7. Lee, L.S.H.; Jim, C.Y. Applying precision triaxial accelerometer to monitor branch sway of an urban tree in a tropical cyclone. Landsc. Urban Plan.; 2018; 178, pp. 170-182. [DOI: https://dx.doi.org/10.1016/j.landurbplan.2018.05.028]

8. Fraser, A.I. The soil and roots as factors in tree stability. Forestry; 1962; 35, pp. 117-127. [DOI: https://dx.doi.org/10.1093/forestry/35.2.117]

9. Schindler, D.; Kolbe, S. Assessment of the Response of a Scots Pine Tree to Effective Wind Loading. Forests; 2020; 11, 145. [DOI: https://dx.doi.org/10.3390/f11020145]

10. Nikolas, A.; Ebba, D.; Jakob, M. Wind load estimation on an open-grown European oak tree. For. Int. J. For. Res.; 2019; 92, pp. 381-392.

11. Dupont, S.; Defossez, P.; Bonnefond, J.M. How stand tree motion impacts wind dynamics during windstorms. Agric. For. Meteorol.; 2018; 262, pp. 42-58. [DOI: https://dx.doi.org/10.1016/j.agrformet.2018.06.022]

12. Van Emmerik, T.; Steele-Dunne, S.; Hut, R.; Gentine, P.; Guerin, M.; Oliveira, R.S.; Wagner, J.; Selker, J.; Van de Giesen, N. Measuring Tree Properties and Responses Using Low-Cost Accelerometers. Sensors; 2017; 17, 1098. [DOI: https://dx.doi.org/10.3390/s17051098] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28492477]

13. Gougherty, A.V.; Keller, S.R.; Kruger, A.; Stylinski, C.D.; Elmore, A.J.; Fitzpatrick, M.C. Estimating tree phenology from high frequency tree movement data. Agric. For. Meteorol.; 2018; 263, pp. 217-224. [DOI: https://dx.doi.org/10.1016/j.agrformet.2018.08.020]

14. Castro-Garcia, S.; Aragon-Rodriguez, F.; Arias-Calderón, R.; Sola-Guirado, R.R.; Gil-Ribes, J.A. The contribution of fruit and leaves to the dynamic response of secondary branches of orange trees. Biosyst. Eng.; 2020; 193, pp. 149-156. [DOI: https://dx.doi.org/10.1016/j.biosystemseng.2020.02.019]

15. Roodbaraky, H.J.; Baker, C.J.; Dawson, A.R.; Wright, C.J. Experimental observations of the aerodynamic characteristics of urban trees. J. Wind. Eng. Ind. Aerodyn.; 1994; 52, pp. 171-184. [DOI: https://dx.doi.org/10.1016/0167-6105(94)90046-9]

16. Schindler, D.; Mohr, M. No resonant response of Scots pine trees to wind excitation. Agric. For. Meteorol.; 2019; 265, pp. 227-244. [DOI: https://dx.doi.org/10.1016/j.agrformet.2018.11.021]

17. Baker, C.J. Measurements of the natural frequencies of trees. J. Exp. Bot.; 1997; 48, pp. 1125-1132. [DOI: https://dx.doi.org/10.1093/jxb/48.5.1125]

18. Hassinen, A.; Lemettinen, M.; Peltola, H.; Kellomäki, S.; Gardiner, B. A prism-based system for monitoring the swaying of trees under wind loading. Agric. For. Meteorol.; 1998; 90, pp. 187-194. [DOI: https://dx.doi.org/10.1016/S0168-1923(98)00052-5]

19. Barbacci, A.; Diener, J.; Hémon, P.; Adam, B.; Donès, N.; Reveret, L.; Moulia, B. A robust videogrametric method for the velocimetry of wind-induced motion in trees. Agric. For. Meteorol.; 2014; 184, pp. 220-229. [DOI: https://dx.doi.org/10.1016/j.agrformet.2013.10.003]

20. Hu, S.; He, P.; He, D. Motion Capture and Estimation of Dynamic Properties for Realistic Tree Animation. Next Generation Computer Animation Techniques; Springer International Publishing: New York, NY, USA, 2017; pp. 18-34.

21. Lucas, B.D. An Iterative Image Registration Technique with an Application to Stereo Vision (DARPA). Proc. Ijcai.; 1981; 3, pp. 674-679.

22. Bouguet, J.Y. Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the Algorithm; Intel Corporation Microprocessor Research Labs: Santa Clara, CA, USA, 2000.

23. Diener, J.; Reveret, L.; Fiume, E. Hierarchical retargetting of 2D motion fields to the animation of 3D plant models. Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA ’06); Vienna, Austria, 2–4 September 2006; pp. 187-195.

24. Der Loughian, C.; Tadrist, L.; Allain, J.M.; Diener, J.; Moulia, B.; de Langre, E. Measuring local and global vibration modes in model plants. Comptes Rendus Mécanique; 2014; 342, pp. 1-7. [DOI: https://dx.doi.org/10.1016/j.crme.2013.10.010]

25. Shi, J.; Tomasi, C. Good features to track. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR; Seattle, WA, USA, 21–23 June 1994; pp. 593-600.

26. Xue, T.; Wu, J.; Zhang, Z.; Zhang, C.; Tenenbaum, J.B.; Freeman, W.T. Seeing Tree Structure from Vibration. Computer Vision. Proceedings of the European Conference on Computer Vision (ECCV); Munich, Germany, 8–14 September 2018; pp. 762-779.

27. Nakata, M.T.; Takahara, M.; Sakamoto, S.; Yoshida, K.; Mitsuda, N. High-throughput analysis of Arabidopsis stem vibrations to identify mutants with altered mechanical properties. Front. Plant Sci.; 2018; 9, 780. [DOI: https://dx.doi.org/10.3389/fpls.2018.00780] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29946329]

28. De Langre, E.; Penalver, O.; Hémon, P.; Frachisse, J.; Bogeat-Triboulot, M.; Niez, B.; Moulia, B. Nondestructive and fast vibration phenotyping of plants. Plant Phenomics; 2019; 2019, 6379693. [DOI: https://dx.doi.org/10.34133/2019/6379693] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33313534]

29. Bolme, D.S.; Beveridge, J.R.; Draper, B.A.; Lui, Y.M. Visual object tracking using adaptive correlation filters. Proceedings of the Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition; San Francisco, CA, USA, 13–18 June 2010; pp. 13-18.

30. Liu, S.; Liu, D.; Srivastava, G.; Połap, D.; Woźniak, M. Overview and methods of correlation filter algorithms in object tracking. Complex Intell. Syst.; 2021; 7, pp. 1895-1917. [DOI: https://dx.doi.org/10.1007/s40747-020-00161-4]

31. Chaudhary, S.; Dudhane, A.; Patil, P.W.; Murala, S.; Talbar, S. Motion estimation in hazy videos. Pattern Recognit. Lett.; 2021; 150, pp. 130-138. [DOI: https://dx.doi.org/10.1016/j.patrec.2021.06.029]