TDMSANet: A Tri-Dimensional Multi-Head Self-Attention Network for Improved Crop Classification from Multitemporal Fine-Resolution Remotely Sensed Images

Abstract

Accurate and timely crop distribution data are crucial for governments, in order to make related policies to ensure food security. However, agricultural ecosystems are spatially and temporally dynamic systems, which poses a great challenge for accurate crop mapping using fine spatial resolution (FSR) imagery. This research proposed a novel Tri-Dimensional Multi-head Self-Attention Network (TDMSANet) for accurate crop mapping from multitemporal fine-resolution remotely sensed images. Specifically, three sub-modules were designed to extract spectral, temporal, and spatial feature representations, respectively. All three sub-modules adopted a multi-head self-attention mechanism to assign higher weights to important features. In addition, the positional encoding was adopted by both temporal and spatial submodules to learn the sequence relationships between the features in a feature sequence. The proposed TDMSANet was evaluated on two sites utilizing FSR SAR (UAVSAR) and optical (Rapid Eye) images, respectively. The experimental results showed that TDMSANet consistently achieved significantly higher crop mapping accuracy compared to the benchmark models across both sites, with an average overall accuracy improvement of 1.40%, 3.35%, and 6.42% over CNN, Transformer, and LSTM, respectively. The ablation experiments further showed that the three sub-modules were all useful to the TDMSANet, and the Spatial Feature Extraction Module exerted larger impact than the remaining two sub-modules.

1. Introduction

It is projected that food demand will surge by about 50% before 2050 with the rapid growth of the global population [1,2]. Traditional agricultural census data (e.g., area of farmland and crop yield) have been widely employed for agricultural planning and management. However, such data are increasingly unable to meet current requirements since they cannot be accessed in a timely manner (fieldwork often takes a long time). There is a growing demand for timely and accurate agricultural thematic information [3,4]. Accurate spatial distribution data of crop types are, therefore, crucial for predicting regional, national, and global crop yield, which provides the basis for governments to make related policies to ensure food security [5,6]. In addition, research reported that a large amount of animals and plants will face extinction by 2050 because of farmland expansion, which is a great loss of natural ecosystems [7]. As a result, timely monitoring of farmland and accurate mapping of crop types are vitally important for ensuring harmony between humanity and nature.

Compared to conventional field surveys, the remote sensing technique offers numerous advantages, including large spatial coverage, timely monitoring and low cost [8,9,10]. Remote sensing has become as one of the most popular means of crop monitoring [11,12]. With the increasing availability of satellite images, we can now acquire massive amounts of remote sensing data every day, which provides the basis for detailed and accurate crop mapping [13,14]. However, agricultural ecosystems are spatially and temporally dynamic, which often results in high within-class variance and low between-class variance in fine spatial resolution (FSR) images [15,16]. Consequently, accurately mapping crops from FSR imagery remains a challenge [17,18,19].

The emergence of deep learning, especially convolutional neural network (CNN), provides unique opportunities for crop classification using FSR imagery. The deep CNN is capable of exploiting the abundant spatial information contained in FSR imagery through an end-to-end way, which has been widely demonstrated to be beneficial for improved crop mapping [20,21,22]. For example, Zhang et al. mapped paddy rice over the Chinese Dongting Lake using CNN, and achieved a very high overall accuracy of 97.06% [23]. Chamundeeswari proposed an improved CNN model for crop mapping and acquired overall accuracies of over 95% on several public datasets [24]. In addition to spatial information, previous research also demonstrated that using multi-temporal imagery can further improve CNN-based crop mapping accuracy due to the phenological differences between crops [25,26,27].

Though many previous studies have demonstrated that CNN can extract spatial information of remotely sensed imagery, there are still some limitations for CNN. Specifically, CNN adopts a weighted-average approach to extract global texture information from the image, which could potentially reduce its sensitivity to individual pixel features [28,29,30]. In other words, although this approach provides global spatial sensing capability, it reduces the capability of extracting features for single pixels. Previous studies also reported these shortcomings of CNN in spectral feature extraction [31]. Furthermore, if the spectral feature of the center pixel cannot be fully exploited, the sensing capability of homogeneous pixels to the center pixel would be diminished. This may lead to insufficient extraction of spatial information.

The aim of this research is to propose a Tri-Dimensional Multi-head Self-Attention Network (TDMSANet) for crop classification from time series FSR remotely sensed imagery. In the TDMSANet, three sub-modules were specially designed to extract spectral, temporal, and spatial feature representations, respectively. By doing so, the spectral, temporal, and spatial information can be extracted simultaneously from remotely sensed imagery, enabling improved crop mapping. Previous studies have demonstrated the effectiveness of incorporating both temporal and spatial information to improve classification accuracies, as images from different time periods offer varying capabilities to distinguish crops, and the spatial features of different crops can vary significantly [15,16]. All three sub-modules adopted a multi-head self-attention mechanism to assign higher weights to important features, and both temporal and spatial sub-modules employed positional encoding to learn sequence relationships between the features in a feature sequence. The proposed TDMSANet was assessed on two sites utilizing FSR Synthetic Aperture Radar (SAR) and optical images, respectively.

2. Method

In this paper, a novel Triple-Dimensional Multi-head Self-Attention Network (TDMSANet) method was proposed for crop classification. Specifically, TDMSANet leveraged multi-head self-attention mechanisms to comprehensively extract spectral features and explicitly capture temporal features. Additionally, a specialized Spatial Feature Extraction Module was designed in TDMSANet to acquire fully spatial features compared to standard CNNs. The code of TDMSANet is available at https://github.com/txh19971013/TDSMANet.

The architecture of the TDMSANet is presented in Figure 1. The TDMSANet is essentially an image patch-based deep learning model, which takes time-series image patches as model inputs. Each image patch has a shape of $n\times s\times s\times m$, where $n$ represents the length of image time series, $s$ represents the width of image patch, and $m$ represents the number of spectral bands. During the classification process, the input image patch was fed into three different channels, respectively, and the outputted features from each channel were concatenated together for final crop classification (Figure 1). The detailed structure of the three modules (channels) is elaborated as follows.

2.1. Spectral Feature Extraction Module

The data preprocessing process of the Spectral Feature Extraction Module is illustrated in Figure 1a. First, we extracted the central pixel values from each time-series image patch. Second, for each band the extracted values from temporal images were stacked chronologically together. As a result, we acquired a spectral feature sequence with a shape of $m\times n$, where $m$ and $n$ are the length of the feature sequence (i.e., number of spectral bands) and the dimension of each spectral feature (i.e., length of image time series), respectively. Finally, a linear layer was adopted to transform the feature sequence to a shape of $m\times 128$ to increase the dimension of each spectral feature.

The transformed feature sequence was employed as the input of Spectral Feature Extraction Module, the structure of which is shown in Figure 2. As demonstrated by the figure, the module consists of two identical blocks. In each block, a Multi-head Self-Attention (MSA) layer was first adopted in the module. Specifically, a residual connection between the input and output of the MSA layer was employed to prevent the issue of gradient vanishing [32]. A Layer Normalization (LayerNorm) layer was adopted subsequently to enable the model to converge quickly, followed by two consecutive linear layers (with ReLU activation function applied between them). Similar to MSA, a residual connection was also designed for the two linear layers, followed by another LayerNorm layer. Finally, a Max-pooling layer was adopted after the second block to acquire the outputted spectral feature representation with a shape of $1\times 128$.

2.2. Temporal Feature Extraction Module

The data preprocessing procedure of the Temporal Feature Extraction Module is illustrated in Figure 1c. Initially, we extracted the central pixel values from each time-series image patch. Subsequently, we stacked the values of $m$ bands from each image together. This results in a temporal feature sequence with a shape of $n\times m$, where $n$ and $m$ represent the length of the feature sequence (i.e., length of image time series) and the dimension of each temporal feature in the feature sequence (i.e., number of spectral bands), respectively. Finally, a linear layer was employed to map the shape of the feature sequence to $n\times 128$, thereby transforming each temporal feature to a higher-dimensional space.

The transformed features were employed as the input of the Temporal Feature Extraction Module, the structure of which is shown in Figure 3. As observed in the figure, the structure is very similar to that of Spectral Feature Extraction Module. The only difference lies in the head of the module, where positional encoding was adopted to capture the sequential relationships among different temporal features, followed by a LayerNorm layer. Positional encoding is a method originally employed by the Transformer to label the sequential order of features in the feature sequence. It employs sine and cosine functions of varying frequencies to compute absolute positional encoding for each feature [33]. It can be calculated using Equation (1), where $pos$ denotes the position of current feature in a feature sequence, $d$ represents the dimensionality of each positional encoding (i.e., the dimension of each feature in the feature sequence), and $k$ denotes the $k$-th dimension of current feature ($k\in \left\{\mathrm{1,2},\dots ,d\right\}$).

(1)$PositionalEncoding=\left\{\begin{array}{cc}\sin \left(pos/{10000}^{k/d}\right),& k\mathrm{mod}2=0\\ \cos \left(pos/{10000}^{k/d}\right),& k\mathrm{mod}2=1\end{array}\right..$

2.3. Spatial Feature Extraction Module

The data preprocessing procedure is depicted in Figure 1b. First, the extracted time-series image patches were transformed into a spatial feature sequence with a length of $s\times s$, where $s$ represents the width of image patch. Each of the spatial features corresponds to a pixel included in the image patch. The central pixel ${\mathrm{p}}_{\mathrm{c}}$ is highlighted using a red square (Figure 1b). Each spatial feature is a vector with a length of $n\times m$, $n$ denotes the length of image time series, and $m$ represents the number of spectral bands. Second, a linear layer was employed to increase the length of each spatial feature vector from $n\times m$ to 128.

Inspired by the “class token” in the VIT (vision transformer) [34], a Spatial Feature Extraction Module was designed for extracting spatial information from remote sensing images for image classification, as demonstrated by Figure 4. The architecture of this module is very similar to that of the Temporal Feature Extraction Module, including the positional encoding and feature extraction block. The positional encoding was adopted here to learn the positional relationship between the central pixel and other pixels within an image patch. At the end of the module, features located at the central pixel of each patch were extracted especially, and they were taken as the “class token” in the Spatial Feature Extraction Module. By doing so, the most useful spatial information within an image patch can be aggregated and assigned to the “class token”.

2.4. Multi-Head Self-Attention (MSA) Layer

The MSA layer was adopted in all three modules to learn more important features from remotely sensed images. The structure of the MSA layer is demonstrated in Figure 5, with its major steps elaborated in detail, as follows. First, the input vector was split averagely into $h$ sections (attention heads), which were replicated into three copies. Each attention head of each copy was subsequently fed into a linear layer for linear transformation. Second, a dot product operation was performed between the heads in the first copy and the corresponding ones in the second copy. Each value of the calculated matrices was scaled into normalized value, based on which a softmax function was adopted to transform the matrices to weight matrices. A dropout layer was subsequently employed after the softmax function to avoid the over-fitting issue. Third, a dot product operation was implemented again between the derived weight matrices and the corresponding attention heads from the third copy. By doing so, $h$ new attention heads were acquired and then concatenated into a single feature, which was subsequently processed through a linear layer. The final features were, therefore, acquired and outputted from the MSA layer. The calculation process of MSA layer can be expressed using the following three formulas:

(2)$\mathrm{Attention}(x,x,x)=\mathrm{softmax}\left({\displaystyle \frac{x\cdot x^T}{\sqrt{d}}}\right)x,$

(3)$hea{d}_i=Attention\left(x_i{w}_i^q,x_i{w}_i^k,x_i{w}_i^v\right),$

(4)$MSA=concat\left(hea{d}_1,\mathrm{\dots },hea{d}_h\right)w^o,$

where $\mathbf{x}$ denotes the input vector, $\sqrt{d}$ is the scale factor, and $d$ is the dimension of the input vector.

3. Experimental Design and Method

3.1. Study Area

The Sacramento Valley within the Central Valley of California, USA, was chosen as the study area in this research (see Figure 6). The climate of this region is characterized as Mediterranean, with hot summers and wet winters. The average temperature is approximately 16.5 °C, and the annual rainfall is about 437 mm. This region is relatively flat and has fertile soil. The Sacramento River provides abundant water resources, creating favorable growing conditions for diverse crop categories. In this research, we chose two representative sites within this valley, namely S1 and S2. Both sites possess rich and diverse crop classes, making them ideal for testing the capability and generalizability of the proposed method.

In S1, four scenes of L-band Uninhabited Aerial Vehicle Synthetic Aperture Radar (UAVSAR) images acquired on 16/06, 20/07, 29/08, and 03/10 in 2011 were employed for crop mapping. The UAVSAR consists of three linear polarizations, namely HH, HV and VV, with a spatial resolution of five meters. There are 10 dominant crops across this area (see Table 1 in the following section).

In S2, four optical RapidEye images were collected. The acquisitions were dated on 30 May, 10 July, 5 August, and 7 September in 2016, respectively. RapidEye shares the same spatial resolution as UAVSAR (5 m), and it includes five bands, including blue, green, red, red-edge, and near-infrared. The radiometric and geometric corrections were implemented for the acquired images before delivery. There are nine major crops in this area (see Table 1 in the following section).

3.2. Ground Reference

In this study, we adopted the Cropland Data Layer (CDL) as a reference. The CDL is a GIS dataset released by the United States Department of Agriculture, which provides detailed information on the extent and types of crop cultivation in the United States [35]. Due to the high accuracy and stable update frequency (yearly), it has been widely used as reference for crop classification [26,36,37]. Despite this, we also identified some classification errors in the CDL near the boundaries of crop fields. To mitigate these errors, we overlaid our very high spatial resolution images with the CDL and delineated accurate boundaries for crop fields larger than 5 hectares within two sites. These fields were subsequently labeled with reference to the CDL to obtain typical samples. To ensure the independence of training and testing samples, the delineated fields were randomly divided into training polygons and testing polygons, and training and testing samples were collected randomly from training and testing polygons, respectively. Specifically, to avoid the issue of sample imbalance, we obtained an equal number of training samples (150) for each class for model training. To test the mapping accuracy of the classification models, we acquired a number of testing samples for each class that was proportional (1/1000) to the number of pixels covered by its respective testing polygon. In total, 3756 and 3664 samples were acquired for S1 and S2, respectively. The samples across the two sites are described in detail in Table 1.

3.3. Model Parameter Settings

3.3.1. TDMSANet Parameter Settings

For crop classification, we stacked the multitemporal images in temporal sequence and then applied an image standardization step to the stacked imagery. The proposed TDMSANet method takes image patch as model input. To determine the optimal size (width) of image patch, the TDMSANet was implemented with a size varying from 5 to 47 with a step of 6, and 17, and 11 are found to be the optimal sizes (with the highest overall accuracies) for S1 and S2, respectively. During the process of feature extraction, the MSA layer divided each input feature (with a shape of $1\times 128$) into eight sections (eight attention heads), each of which had a feature length of 16. A dropout was adopted in the MSA layer with a probability of 0.1.

To ensure model convergence, the maximum training iterations was set as 500 for the two models on the two sites. In addition, we specified that training terminates if model accuracy does not improve over 100 consecutive iterations to avoid over-fitting. Additionally, to prevent the models from getting stuck in local optima, a cosine annealing with warm restarts algorithm [38] was employed for dynamic learning rate adjustments, with a learning rate range of 0.00025~0.001.

3.3.2. Benchmark Methods and Their Settings

To thoroughly assess the effectiveness of TDMSANet, it was benchmarked against three widely used models: CNN, Transformer, and Long Short-Term Memory (LSTM). Among the four models, CNN and TDMSANet are patch-based models, while Transformer and LSTM are pixel-based models. To make a fair comparison, the settings of the number of maximum iterations and learning rate for the three benchmarks were the same as the TDMSANet.

The hyperparameters for the benchmark models are described as follows:

• (1). CNN: The CNN architecture consists of 4 two-dimensional convolutional layers to prevent overfitting. The initial layer employed a kernel size of 5 × 5, whereas the subsequent convolutional layers utilized a kernel size of 3 × 3. Activation of the convolutional layers was achieved through the ReLU function, followed by a max-pooling layer with a dimension of 3 × 3. The number of features outputted by the four convolutional layers were set as 64, 128, 256, and 512, respectively. A two-layer multilayer perceptron with the number of neurons of 512 was applied at the end of the CNN model for image classification. The optimal image patch sizes for S1 and S2 were found to be 23 and 17, respectively, through cross-validation.

• (2). Transformer: The Transformer model used in this study consists of two encoder layers. The word embedding dimension was set as 128, and the number of attention heads was configured to 8. A dropout probability of 0.1 was applied in the model. A max-pooling layer was designed after the last encoder layer, followed by a two-layer multilayer perceptron with the number of neurons of 512 for image classification, the same as the CNN.

• (3). LSTM: In this study, a LSTM model with two basic bidirectional layers was employed. Since a total of four remotely sensed images are employed at both S1 and S2, the temporal sequence length (lag) was set as 4. The number of features for the hidden layers of the LSTM was set to 256. Dropout was not adopted in the model. The same two-layer multilayer perceptron employed in both CNN and Transformer was placed at the end of the LSTM model for image classification.

3.4. Accuracy Assessment

Based on the classified maps and the reference data, we established the confusion matrices, based on which several accuracy assessment indicators were calculated [39] to quantitatively evaluate the classification accuracy of different classification models. The employed indicators include overall accuracy (OA), recall, precision, and F1 scores (F1). The formulas of the four indicators are as follows:

(5)$OA={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}},$

(6)$recall={\displaystyle \frac{TP}{TP+FN}},$

(7)$precision={\displaystyle \frac{TP}{TP+FP}},$

(8)$F1={\displaystyle \frac{2\times precision\times recall}{precision+recall}},$

where TP is the number of true positives, FP represents the number of false positives, TN is the number of true negatives, and FN indicates the number of false negatives.

3.5. Experimental Results and Analysis

3.5.1. Sensitivity Analysis of Image Patch Size on TDMSANet and CNN

Both TDMSANet and CNN are image patch-based deep learning models, and the image patch size, therefore, markedly affects the performance of the two models [40]. A sensitivity analysis of image patch size on model accuracy was performed. Figure 7 illustrates the variations in overall accuracy achieved by TDMSANet and CNN over the two study sites with the image patch size varying from 5 × 5 to 47 × 47 with a step size of 6. As can be seen from the figure, TDMSANet outperformed CNN prominently across various image patch sizes in both S1 and S2, demonstrating its great capability for crop mapping using image time series. Specifically, the proposed TDMSANet and the CNN exhibited similar trends in OA over both S1 and S2. The OA for both methods boosted significantly when adopting gradually larger image patch sizes, and it started to decrease after reaching the maximum OA with the increase in patch size. The TDMSANet and CNN achieved the highest OAs with patch sizes of 17 × 17 and 23 × 23 in S1, and patch sizes of 11 × 11 and 17 × 17 in S2, respectively. However, the OA of CNN across both study sites decreased rapidly when the patch size exceeded 23 × 23 in S1 and 17 × 17 in S2, which were the patch sizes associated with the highest OAs. In contrast, the OA of TDMSANet decreased relatively slowly when the patch size exceeded 17 × 17 in S1 and 11 × 11 in S2, demonstrating its robustness and insensitivity to variations in image patch sizes.

3.5.2. Model Performance Comparison

Table 2 summarized the crop mapping accuracy attained by the four models across the two study sites. It is evident that the proposed TDMSANet consistently outperformed the benchmark models, with the OAs reaching 86.13% and 90.28% for S1 and S2, surpassing the CNN (84.74% and 88.88%), Transformer (81.69% and 88.03%), and LSTM (77.53% and 86.04%). Additionally, the TDMSANet also achieved the highest values for the other three indicators (i.e., Recall, Precision, and F1) across both sites. For example, the TDMSANet acquired the largest F1 values, reaching 86.18% and 90.31% for S1 and S2, greater than those of the CNN (84.76% and 88.87%), Transformer (81.85% and 88.22%), and LSTM (77.70% and 85.84%), respectively. To test whether there was a significant difference between the proposed method and the benchmarks, a McNemar test designed for pair-wise comparison was conducted across both study sites (Table 3). As can be seen from the table, the proposed TDMSANet consistently acquired significantly better mapping accuracies than the CNN, Transformer, and LSTM, with z-value = 2.14, 5.93, and 10.98 in S1 and z-value = 2.47, 3.62, and 6.96 in S2, respectively (Table 3).

The superiority of TDMSANet can also be observed from the per-class classification accuracy detailed in Table 4. The table illustrates that TDMSANet achieved the highest mapping accuracy across the majority of crop categories in S1 and in four categories in S2, as highlighted by the bold numbers. For S1, the most pronounced increase in accuracy was observed for Clover, which exhibited a remarkable accuracy of 92.98%. This was higher than the CNN, Transformer and LSTM by 17.54%, 9.65% and 19.30%, respectively. Similarly, a marked accuracy (3.45–15.52%) increase was also seen for both Almond and Pepper in comparison to the benchmarks. For S2, the TDMSANet generally achieved significantly higher per-class mapping accuracy compared to the Transformer, LSTM, and CNN. For example, the mapping accuracy for Walnut was 93.94%, higher by 7.58% and 16.16% compared with that of Transformer and LSTM, respectively. The accuracy increases for Fallow were also notable, registering an increase of 11.66% and 23.33% in comparison to Transformer and LSTM, respectively. Similarly, the TDMSANet increased mapping accuracy for most of the crop categories in comparison to CNN. Specifically, the accuracy increase was over 12% for Fallow and Cucumber, and over 2% for Walnut and Tomato.

3.5.3. Crop Mapping Results

The classified maps by the proposed TDMSANet and three benchmarks were compared in S1 (Figure 8) and S2 (Figure 9), respectively, using three typical image subsets. For S1, as illustrated in the figure, the classification maps of the two pixel-based model (Transformer and LSTM) presented heavy salt-and-pepper noise (Figure 8a,c). In addition, the misclassifications between crop classes were also obvious. For example, both models misidentified a large portion of Alfalfa as Wheat (Figure 8c). Furthermore, some linear noises were also observed from the classified maps of the two models (Figure 8a). Compared with the pixel-based models, the two patch-based models (CNN and TDMSANet) achieved more smooth mapping results with little salt-and-pepper noise. The classifications of some crop classes (e.g., Alfalfa and Corn) were also improved to some extent (Figure 8b). However, CNN was unable to identify Pepper and Alfalfa from Tomato and Hay, respectively (Figure 8a,b). Fortunately, the proposed TDMSANet successfully corrected all the mentioned misclassifications, and achieved the most accurate crop mapping results compared with the other three models. For S2, the three benchmarks were found difficult to differentiate Tomato from Sunflower (Figure 9a,c). Moreover, they often misclassified other crops as Wheat (Figure 9b,c). Similar to S1, the proposed TDMSANet revised those classification errors and, thus, acquired desirable crop mapping results.

3.5.4. Computational Efficiency of the Models

In addition to mapping accuracy, computational efficiency is another critical indicator for evaluating the performance of deep learning models. We, therefore, summarized the model size, training time and testing time (for each epoch) for the proposed TDMSANet as well as three benchmarks (CNN, Transformer, and LSTM) in Table 5. As shown in the table, the training time for the proposed TDMSANet model is significantly higher than that of the benchmark models across both sites. This is primarily due to the use of the MSA layer, whose computational complexity increases exponentially with the size of image patch. Fortunately, since there is no need for backpropagation or saving intermediate variables during the model inference process, the testing time of TDMSANet is not significantly increased compared with other models (especially CNN). Additionally, the model size of TDMSANet is substantially smaller than that of CNN and LSTM across both sites, indicating that TDMSANet requires significantly less memory. Therefore, by appropriately increasing the batch size, TDMSANet can achieve testing times comparable to CNN and LSTM.

3.5.5. Ablation Experiment

To validate the usefulness of each module for the proposed TDMSANet, the impact of the Spectral Feature Extraction Module, Temporal Feature Extraction Module, and Spatial Feature Extraction Module on the performance of TDMSANet was assess over both S1 and S2. Specifically, we constructed three new comparative models by removing each of these three modules individually: TDMSANet_noSpe, TDMSANet_noT, and TDMSANet_noSpa. Table 6 presents the mapping accuracy of the comparative models as well as the TDMSANet in both sites. As demonstrated in the table, the overall mapping accuracy decreased by about 1% for S1 when removing the Spectral Feature Extraction Module or Temporal Feature Extraction Module. However, the overall mapping accuracy decreased markedly by about 5% without use of the Spatial Feature Extraction Module. The other three accuracy indicators (Recall, Precision, and F1) showed similar tendencies to OA. The accuracy decrease tendency in S2 was similar to that in S1. Specifically, the overall mapping accuracy decreased by about 1.5% without using the Spectral Feature Extraction Module or Temporal Feature Extraction Module. When removed the Spatial Feature Extraction Module, the mapping accuracy decreased by around 3%. In other words, the three modules are all useful to the TDMSANet, and the Spatial Feature Extraction Module exerts larger impact than the remaining two modules.

In the proposed TDMSANet model, positional encoding was incorporated into both the Temporal Feature Extraction and Spatial Feature Extraction modules to enhance temporal and spatial feature extraction capabilities. To validate the effectiveness of positional encoding, we conducted a stepwise analysis. First, we removed positional encoding from both modules, resulting in a new model named TDMSANet_noPos. Next, we added positional encoding to the Temporal Feature Extraction module, creating a model named TDMSANet_tPos. Finally, we included positional encoding in both modules to establish the full TDMSANet model.

We tested these models on S1 and S2 datasets and summarized the results in Table 7. As shown, TDMSANet_noPos, without positional encoding, achieved an overall accuracy (OA) of 84.35% on S1 and 88.72% on S2. When positional encoding was added to the Temporal Feature Extraction module, the OA increased to 85.11% (+0.76%) on S1 and 89.37% (+0.65%) on S2. Finally, the highest OAs of 86.13% and 90.28% were achieved on S1 and S2, respectively, when positional encoding was included in both Temporal and Spatial Feature Extraction modules. Notably, the other three accuracy metrics exhibited a similar trend to OA across the two sites.

4. Discussion

Agricultural ecosystems are highly dynamic systems which are susceptible to climate change and human activities. Therefore, agricultural landscapes are usually complex and heterogeneous. Multitemporal FSR remote sensing images provides unique opportunities for accurate mapping of agricultural landscape. However, how to fully exploit the rich information in FSR remains a big challenge. Some previous studies have designed several CNN-based models with the aim of achieving desirable crop mapping results. For example, Wang et al. proposed a new attention-based CNN model for large-scale crop mapping using FSR image time series [41]. Ji et al. classified crop types using 3D CNN with multi-temporal FSR images [42]. Some research also exploited simultaneously spatial, temporal, and spectral features from remotely sensed imagery for crop classification [43,44]. However, these previous methods were generally evolved from standard CNNs, which have shortcomings in paying attention to important input variables [45,46]. Such limitations make them difficult to fully exploit the rich features in FSR image time series. In this research, we, therefore, abandoned the CNN structure, and proposed a novel deep learning model based entirely on multi-head self-attention layers for crop classification. In the proposed TDMSANet, three sub-modules with multi-head self-attention were designed specially for extracting spectral, temporal and spatial features from multitemporal FSR images. As a result, important input variables in each sub-module can be assigned more weight, enabling the rich spectral, temporal, and spatial features to be fully learned from remotely sensed imagery. The consistently better crop mapping results achieved by TDMSANet have demonstrated its significant superiority over the CNN model. In addition to deep learning benchmarks, we also implemented conventional Random Forest for accuracy comparison [15,16]. The achieved OAs by RF over the two study sites were 72.92% and 84.83%, respectively, which were far lower than those of the proposed TDMSANet and other three deep learning benchmarks.

The ablation experiments demonstrated the effectiveness of all three sub-modules. This also explains why the proposed TDMSANet consistently outperformed CNN in both sites. As expected, ablation experiments revealed that the spatial features are much more important than the other two types of features for crop mapping, which aligns with previous studies [42]. In the proposed TDMSANet, we specially designed Spatial Feature Extraction Module for spatial feature extraction. Both positional encoding and a multi-head self-attention mechanism were adopted in the module, with the aim of ensuring that pixels more similar to the center pixel of the image patch receive more attention during the mining process of spatial information. To demonstrate the superiority of the multi-head self-attention in spatial feature extraction, we compared the weight maps between the proposed TDMSANet and CNN. Specifically, four pixels located near the boundary between crop fields and their corresponding image patches were shown in Figure 10. The weights of each pixel within each image patch achieved by TDMSANet and CNN were generated using the Ablation-cam algorithm [47] and demonstrated in Figure 10a–h. As depicted in the figure, TDMSANet exhibits heightened attention (highlighted pixels in yellow) towards the right-lower, upper, left-lower, and lower regions for the green rectangles in Figure 10E–H, respectively. Notably, this attention pattern spatially corresponds to the distribution of the target crop fields (depicted in pink). Conversely, it exhibited reduced attention (as indicated by pixels in dark blue) towards the non-target crop fields and background. In other words, the Spatial Feature Extraction Module effectively filtered out noise and background information during the feature extraction process through positional encoding and multi-head self-attention mechanism. On the contrary, CNN primarily allocated high attention to only a few pixels within the target fields. This indicates that the CNN did not fully exploit the rich spatial information present in the FSR images.

As demonstrated in the results section, the crop mapping accuracy achieved by both CNN and TDMSANet initially increased, reached a peak, and then decreased as the patch size grew. This is consistent with the reports by previous studies [1,40]. This is because deep learning models may not extract sufficient information from imagery for classification if the patch size is too small, while an excessive amount of noise may be included if the patch size is too large. Despite this, we observed that the optimal patch sizes of TDMSANet (11 × 11 and 17 × 17) are smaller than that of CNN (17 × 17 and 23 × 23) across both sites. This may be attributed to the strong feature extraction capability of TDMSANet, which can learn sufficient information for accurate crop classification using smaller patch sizes compared to CNN. Furthermore, we also observed that TDMSANet performed more stably than CNN across the patch sizes over both sites (Figure 7). For example, in S1, the difference between the highest and lowest overall accuracies was about 3% for the TDMSANet, whereas it was 7% for the CNN. The proposed TDMSANet, fully constituted of multi-head self-attention, possesses the capability of assigning high weights to important features (i.e., assigning low weights to noise). As a result, it can resist noise when the adopted patch sizes exceed the optimal size and achieve relatively accurate mapping results (Figure 7).

5. Conclusions

This study proposed a novel Tri-Dimensional Multi-head Self-Attention Network (TDMSANet) approach for image patch-based crop mapping using multi-temporal FSR remote sensing images. The effectiveness of the TDMSANet was comprehensively evaluated using both optical and radar FSR images. The experimental results demonstrated that the presented method significantly outperformed the other three benchmarks (CNN, Transformer and LSTM) for crop mapping. The ablation experiments validated the utility of the three modules designed for spectral, temporal, and spatial information extraction. Specifically, compared with CNN, we observed that the TDMSANet assigned higher weights to the target fields within image patch. In other words, the TDMSANet can fully exploit the abundant information contained in the FSR imagery. We, therefore, conclude that the newly developed TDMSANet is an effective approach for crop mapping using multi-temporal FSR remote sensing imagery.

Author Contributions

Conceptualization, J.L. (Jian Li) and H.L.; Methodology, X.T.; Software, X.T.; Validation, X.T.; Investigation, X.T., J.L. (Jian Lu), H.F. and J.H.; Writing—original draft, X.T.; Writing—review & editing, J.L. (Jian Li), M.Z., C.Z. and H.L.; Visualization, X.T.; Supervision, J.L. (Jian Li) and H.L.; Project administration, J.L. (Jian Li) and H.L. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Architecture of the TDMSANet. (a) Spectral Feature Extraction Module, (b) Spatial Feature Extraction Module, and (c) Temporal Feature Extraction Module.

Figure 2. The architecture of Spectral Feature Extraction Module.

Figure 3. The architecture of the Temporal Feature Extraction Module.

Figure 4. The architecture of Spatial Feature Extraction Module.

Figure 5. The architecture of MSA layer.

Figure 6. Geographical locations and ground reference of the two study sites in California, USA.

Figure 7. The impact of image patch size on the overall accuracy of TDMSANet and CNN over both sites.

Figure 8. Three representative image subsets of S1 (a–c) and their classified maps with different methods.

Figure 9. Three representative image subsets of S2 (a–c) and their classified maps with different methods.

Figure 10. Typical image subsets from the two sites and their corresponding weight maps achieved by CNN and the proposed TDMSANet. (A–D) typical image subsets from the two sites, (E–H) the corresponding reference maps (target and non-target fields are highlighted in pink and yellow colors, respectively), (a–d) the corresponding weight maps within an image patch (green rectangles in the Reference maps) by TDMSANet, (e–h) the corresponding weight maps within an image patch (green rectangles in the Reference maps) by CNN. Note that each blue dot in subfigures (E–H) indicates the center pixel of a convolution patch.

Table 1

Detailed information about the crop categories and sample points for S1 and S2.

Site	Category	Train Sample	Test Sample	Total Sample
S1	Walnut	150	121	271
	Almond	150	94	244
	Alfalfa	150	514	664
	Hay	150	127	277
	Clover	150	114	264
	Wheat	150	440	590
	Corn	150	152	302
	Sunflower	150	184	334
	Tomato	150	452	602
	Pepper	150	58	208
	Total	1500	2256	3756
S2	Walnut	150	198	348
	Almond	150	259	409
	Fallow	150	60	210
	Alfalfa	150	249	399
	Wheat	150	277	427
	Corn	150	62	212
	Sunflower	150	455	605
	Tomato	150	671	821
	Cucumber	150	83	233
	Total	1350	2314	3664

Table 2

Classification accuracy achieved by different models over both sites. Note that the highest value in each column is highlighted in bold.

Site	Model	OA	Recall	Precision	F1
S1	TDMSANet	86.13	86.13	86.86	86.18
	CNN	84.74	84.74	85.09	84.76
	Transformer	81.69	81.69	83.01	81.85
	LSTM	77.53	77.53	78.78	77.70
S2	TDMSANet	90.28	90.28	90.53	90.31
	CNN	88.88	88.88	88.92	88.87
	Transformer	88.03	88.03	88.66	88.22
	LSTM	86.04	86.04	86.34	85.84

Table 3

Mcnemar test by comparing the proposed TDMSANet with the benchmarks. Note that z-value > 1.96 indicates significant difference at the 95% confidence level, and is highlighted in bold.

Site	Model	McNemar Test z-Value
		TDMSANet	CNN	Transformer	LSTM
S1	TDMSANet	-
	CNN	2.14	-
	Transformer	5.93	3.54	-
	LSTM	10.98	8.52	5.71	-
S2	TDMSANet	-
	CNN	2.47	-
	Transformer	3.62	1.32	-
	LSTM	6.96	4.69	3.24	-

Table 4

Per-class classification accuracy comparison amongst TDMSANet, CNN, Transformer, and LSTM on the two sites. Note that the highest value in each line is highlighted in bold.

Site.	Category	TDMSANet	CNN	Transformer	LSTM
S1	Walnut	96.69	95.04	90.08	76.03
	Almond	93.62	89.25	86.17	84.04
	Alfalfa	83.27	79.77	69.65	74.90
	Hay	77.17	75.59	69.29	70.08
	Clover	92.98	75.44	83.33	73.68
	Wheat	71.59	72.44	79.77	66.59
	Corn	94.74	96.71	90.13	77.63
	Sunflower	83.70	89.13	83.15	71.20
	Tomato	96.46	96.90	92.04	95.13
	Pepper	98.28	91.38	94.83	82.76
S2	Walnut	93.94	90.91	86.36	77.78
	Almond	81.85	86.10	79.54	78.38
	Fallow	68.33	48.33	56.67	45.00
	Alfalfa	88.35	93.17	92.37	85.14
	Wheat	97.11	96.75	93.14	96.75
	Corn	96.77	98.39	95.16	96.77
	Sunflower	92.09	90.11	93.63	92.09
	Tomato	95.53	92.68	88.82	92.70
	Cucumber	50.60	38.55	68.67	30.12

Table 5

Model size and computational efficiency comparison amongst different models.

Site	Model	Model Size	Training Time (s)	Testing Time (s)
S1	TDMSANet	951,434	0.96	0.57
	CNN	1,836,234	0.21	0.29
	Transformer	283,530	0.22	0.18
	LSTM	3,165,706	0.22	0.21
S2	TDMSANet	952,329	0.47	0.54
	CNN	1,848,521	0.16	0.43
	Transformer	283,657	0.21	0.21
	LSTM	3,169,289	0.22	0.28

Table 6

Accuracy comparison amongst different models in the ablation experiments. Note that the highest value in each column is highlighted in bold.

Site	Model	OA	Recall	Precision	F1
S1	TDMSANet	86.13	86.13	86.86	86.18
	TDMSANet_noSpe	85.28	85.28	86.38	85.50
	TDMSANet_noT	85.20	85.20	86.11	85.06
	TDMSANet_noSpa	81.16	81.16	82.94	81.30
S2	TDMSANet	90.28	90.28	90.53	90.31
	TDMSANet_noSpe	88.76	88.76	88.38	88.49
	TDMSANet_noT	88.59	88.59	88.93	88.60
	TDMSANet_noSpa	87.25	87.25	87.85	87.30

Table 7

Accuracy comparison amongst different models to test the effectiveness of positional encoding. Note that the highest value in each column is highlighted in bold.

Site	Model	OA	Recall	Precision	F1
S1	TDMSANet_noPos	84.35	84.35	85.21	84.50
	TDMSANet_tPos	85.11	85.11	86.13	85.38
	TDMSANet	86.13	86.13	86.86	86.18
S2	TDMSANet_noPos	88.72	88.72	88.21	88.35
	TDMSANet_tPos	89.37	89.37	89.45	89.36
	TDMSANet	90.28	90.28	90.53	90.31

References

1. Li, H.; Zhang, C.; Zhang, Y.; Zhang, S.; Ding, X.; Atkinson, P.M. A Scale Sequence Object-based Convolutional Neural Network (SS-OCNN) for crop classification from fine spatial resolution remotely sensed imagery. Int. J. Digit. Earth; 2021; 14, pp. 1528-1546. [DOI: https://dx.doi.org/10.1080/17538947.2021.1950853]

2. Falcon, W.P.; Naylor, R.L.; Shankar, N.D. Rethinking global food demand for 2050. Popul. Dev. Rev.; 2022; 48, pp. 921-957. [DOI: https://dx.doi.org/10.1111/padr.12508]

3. Huo, D.; Malik, A.W.; Ravana, S.D.; Rahman, A.U.; Ahmedy, I. Mapping smart farming: Addressing agricultural challenges in data-driven era. Renew. Sustain. Energy Rev.; 2024; 189, 113858. [DOI: https://dx.doi.org/10.1016/j.rser.2023.113858]

4. Li, H.; Huang, J.; Zhang, C.; Ning, X.; Zhang, S.; Atkinson, P.M. An efficient and generalisable approach for mapping paddy rice fields based on their unique spectra during the transplanting period leveraging the CIE colour space. Remote Sens. Environ.; 2024; 313, 114381. [DOI: https://dx.doi.org/10.1016/j.rse.2024.114381]

5. Mutanga, O.; Dube, T.; Galal, O. Remote sensing of crop health for food security in Africa: Potentials and constraints. Remote Sens. Appl. Soc. Environ.; 2017; 8, pp. 231-239. [DOI: https://dx.doi.org/10.1016/j.rsase.2017.10.004]

6. Liao, X.; Yao, Q.; Wan, X.; Wang, J.; Li, Z. Theoretical basis and technical path for the regional all-for-one customization model of black soil granary. J. Geogr. Sci.; 2022; 32, pp. 2147-2169. [DOI: https://dx.doi.org/10.1007/s11442-022-2041-2]

7. Williams, D.R.; Clark, M.; Buchanan, G.M.; Ficetola, G.F.; Rondinini, C.; Tilman, D. Proactive conservation to prevent habitat losses to agricultural expansion. Nat. Sustain.; 2021; 4, pp. 314-322. [DOI: https://dx.doi.org/10.1038/s41893-020-00656-5]

8. Jin, X.; Li, Z.; Feng, H.; Ren, Z.; Li, S. Deep neural network algorithm for estimating maize biomass based on simulated Sentinel 2A vegetation indices and leaf area index. Crop J.; 2020; 8, pp. 87-97. [DOI: https://dx.doi.org/10.1016/j.cj.2019.06.005]

9. Jin, X.; Li, Z.; Yang, G.; Yang, H.; Feng, H.; Xu, X.; Wang, J.; Li, X.; Luo, J. Winter wheat yield estimation based on multi-source medium resolution optical and radar imaging data and the AquaCrop model using the particle swarm optimization algorithm. ISPRS J. Photogramm. Remote Sens.; 2017; 126, pp. 24-37. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2017.02.001]

10. Lu, J.; Fu, H.; Tang, X.; Liu, Z.; Huang, J.; Zou, W.; Chen, H.; Sun, Y.; Ning, X.; Li, J. GOA-optimized deep learning for soybean yield estimation using multi-source remote sensing data. Sci. Rep.; 2024; 14, 7097. [DOI: https://dx.doi.org/10.1038/s41598-024-57278-6] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/38528045]

11. Omia, E.; Bae, H.; Park, E.; Kim, M.S.; Baek, I.; Kabenge, I.; Cho, B.-K. Remote sensing in field crop monitoring: A comprehensive review of sensor systems, data analyses and recent advances. Remote Sens.; 2023; 15, 354. [DOI: https://dx.doi.org/10.3390/rs15020354]

12. Jafarbiglu, H.; Pourreza, A. A comprehensive review of remote sensing platforms, sensors, and applications in nut crops. Comput. Electron. Agric.; 2022; 197, 106844. [DOI: https://dx.doi.org/10.1016/j.compag.2022.106844]

13. Gao, F.; Zhang, X. Mapping crop phenology in near real-time using satellite remote sensing: Challenges and opportunities. J. Remote Sens.; 2021; 2021, 8379391. [DOI: https://dx.doi.org/10.34133/2021/8379391]

14. Burke, M.; Driscoll, A.; Lobell, D.B.; Ermon, S. Using satellite imagery to understand and promote sustainable development. Science; 2021; 371, eabe8628. [DOI: https://dx.doi.org/10.1126/science.abe8628] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33737462]

15. Li, H.; Tian, Y.; Zhang, C.; Zhang, S.; Atkinson, P.M. Temporal Sequence Object-based CNN (TS-OCNN) for crop classification from fine resolution remote sensing image time-series. Crop J.; 2022; 10, pp. 1507-1516. [DOI: https://dx.doi.org/10.1016/j.cj.2022.07.005]

16. Chen, H.; Li, H.; Liu, Z.; Zhang, C.; Zhang, S.; Atkinson, P.M. A novel Greenness and Water Content Composite Index (GWCCI) for soybean mapping from single remotely sensed multispectral images. Remote Sens. Environ.; 2023; 295, 113679. [DOI: https://dx.doi.org/10.1016/j.rse.2023.113679]

17. Li, H.; Zhang, C.; Zhang, S.; Ding, X.; Atkinson, P.M. Iterative Deep Learning (IDL) for agricultural landscape classification using fine spatial resolution remotely sensed imagery. Int. J. Appl. Earth Obs. Geoinf.; 2021; 102, 102437. [DOI: https://dx.doi.org/10.1016/j.jag.2021.102437]

18. Fu, Y.; Ye, Z.; Deng, J.; Zheng, X.; Huang, Y.; Yang, W.; Wang, Y.; Wang, K. Finer resolution mapping of marine aquaculture areas using worldView-2 imagery and a hierarchical cascade convolutional neural network. Remote Sens.; 2019; 11, 1678. [DOI: https://dx.doi.org/10.3390/rs11141678]

19. de Azevedo, R.P.; Dallacort, R.; Boechat, C.L.; Teodoro, P.E.; Teodoro, L.P.R.; Rossi, F.S.; Correia Filho, W.L.F.; Della-Silva, J.L.; Baio, F.H.R.; Lima, M. Remotely sensed imagery and machine learning for mapping of sesame crop in the Brazilian Midwest. Remote Sens. Appl. Soc. Environ.; 2023; 32, 101018. [DOI: https://dx.doi.org/10.1016/j.rsase.2023.101018]

20. Bhosle, K.; Musande, V. Evaluation of deep learning CNN model for land use land cover classification and crop identification using hyperspectral remote sensing images. J. Indian Soc. Remote. Sens.; 2019; 47, pp. 1949-1958. [DOI: https://dx.doi.org/10.1007/s12524-019-01041-2]

21. Lin, C.; Jin, Z.; Mulla, D.; Ghosh, R.; Guan, K.; Kumar, V.; Cai, Y. Toward large-scale mapping of tree crops with high-resolution satellite imagery and deep learning algorithms: A case study of olive orchards in Morocco. Remote Sens.; 2021; 13, 1740. [DOI: https://dx.doi.org/10.3390/rs13091740]

22. Haq, M.A. CNN based automated weed detection system using UAV imagery. Comput. Syst. Sci. Eng.; 2022; 42, pp. 837-849.

23. Zhang, M.; Lin, H.; Wang, G.; Sun, H.; Fu, J. Mapping paddy rice using a convolutional neural network (CNN) with Landsat 8 datasets in the Dongting Lake Area, China. Remote Sens.; 2018; 10, 1840. [DOI: https://dx.doi.org/10.3390/rs10111840]

24. Chamundeeswari, G.; Srinivasan, S.; Bharathi, S.P.; Priya, P.; Kannammal, G.R.; Rajendran, S. Optimal deep convolutional neural network based crop classification model on multispectral remote sensing images. Microprocess. Microsyst.; 2022; 94, 104626. [DOI: https://dx.doi.org/10.1016/j.micpro.2022.104626]

25. Zhong, L.; Hu, L.; Zhou, H. Deep learning based multi-temporal crop classification. Remote Sens. Environ.; 2019; 221, pp. 430-443. [DOI: https://dx.doi.org/10.1016/j.rse.2018.11.032]

26. Xu, J.; Zhu, Y.; Zhong, R.; Lin, Z.; Xu, J.; Jiang, H.; Huang, J.; Li, H.; Lin, T. DeepCropMapping: A multi-temporal deep learning approach with improved spatial generalizability for dynamic corn and soybean mapping. Remote Sens. Environ.; 2020; 247, 111946. [DOI: https://dx.doi.org/10.1016/j.rse.2020.111946]

27. Jiao, X.; Kovacs, J.M.; Shang, J.; McNairn, H.; Walters, D.; Ma, B.; Geng, X. Object-oriented crop mapping and monitoring using multi-temporal polarimetric RADARSAT-2 data. ISPRS J. Photogramm. Remote Sens.; 2014; 96, pp. 38-46. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2014.06.014]

28. Byra, M.; Galperin, M.; Ojeda-Fournier, H.; Olson, L.; O’Boyle, M.; Comstock, C.; Andre, M. Breast mass classification in sonography with transfer learning using a deep convolutional neural network and color conversion. Med. Phys.; 2019; 46, pp. 746-755. [DOI: https://dx.doi.org/10.1002/mp.13361] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30589947]

29. Hsu, W.-Y.; Lin, W.-Y. Ratio-and-scale-aware YOLO for pedestrian detection. IEEE Trans. Image Process.; 2020; 30, pp. 934-947. [DOI: https://dx.doi.org/10.1109/TIP.2020.3039574]

30. Xie, Y.; Richmond, D. Pre-training on Grayscale ImageNet Improves Medical Image Classification. Proceedings of the Computer Vision–ECCV 2018 Workshops; Munich, Germany, 8–14 September 2018; Proceedings, Part VI 15 Springer: Berlin/Heidelberg, Germany, 2019; pp. 476-484.

31. Chen, H.; Yin, D.; Chen, J.; Chen, X.; Liu, S.; Liu, L. Stacked spectral feature space patch: An advanced spectral representation for precise crop classification based on convolutional neural network. Crop J.; 2022; 10, pp. 1460-1469. [DOI: https://dx.doi.org/10.1016/j.cj.2021.12.011]

32. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. Proceedings of the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770-778.

33. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst.; 2017; 30, pp. 5998-6008.

34. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv; 2020; arXiv: 2010.11929

35. Boryan, C.; Yang, Z.; Mueller, R.; Craig, M. Monitoring US agriculture: The US department of agriculture, national agricultural statistics service, cropland data layer program. Geocarto Int.; 2011; 26, pp. 341-358. [DOI: https://dx.doi.org/10.1080/10106049.2011.562309]

36. Hao, P.; Di, L.; Zhang, C.; Guo, L. Transfer Learning for Crop classification with Cropland Data Layer data (CDL) as training samples. Sci. Total Environ.; 2020; 733, 138869. [DOI: https://dx.doi.org/10.1016/j.scitotenv.2020.138869]

37. Lark, T.J.; Schelly, I.H.; Gibbs, H.K. Accuracy, bias, and improvements in mapping crops and cropland across the United States using the USDA cropland data layer. Remote Sens.; 2021; 13, 968. [DOI: https://dx.doi.org/10.3390/rs13050968]

38. Loshchilov, I.; Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. Proceedings of the International Conference on Learning Representations; San Juan, Puerto Rico, 2–4 May 2016.

39. Liu, C.; Frazier, P.; Kumar, L. Comparative assessment of the measures of thematic classification accuracy. Remote Sens. Environ.; 2007; 107, pp. 606-616. [DOI: https://dx.doi.org/10.1016/j.rse.2006.10.010]

40. Zhang, C.; Sargent, I.; Pan, X.; Li, H.; Gardiner, A.; Hare, J.; Atkinson, P.M. An object-based convolutional neural network (OCNN) for urban land use classification. Remote Sens. Environ.; 2018; 216, pp. 57-70. [DOI: https://dx.doi.org/10.1016/j.rse.2018.06.034]

41. Wang, Y.; Zhang, Z.; Feng, L.; Ma, Y.; Du, Q. A new attention-based CNN approach for crop mapping using time series Sentinel-2 images. Comput. Electron. Agric.; 2021; 184, 106090. [DOI: https://dx.doi.org/10.1016/j.compag.2021.106090]

42. Ji, S.; Zhang, C.; Xu, A.; Shi, Y.; Duan, Y. 3D convolutional neural networks for crop classification with multi-temporal remote sensing images. Remote Sens.; 2018; 10, 75. [DOI: https://dx.doi.org/10.3390/rs10010075]

43. Luo, C.; Meng, S.; Hu, X.; Wang, X.; Zhong, Y. Cropnet: Deep spatial-temporal-spectral feature learning network for crop classification from time-series multi-spectral images. Proceedings of the IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium; Waikoloa, HI, USA, 26 September–2 October 2020; pp. 4187-4190.

44. He, D.; Zhong, Y.; Zhang, L. Spectral–spatial–temporal MAP-based sub-pixel mapping for land-cover change detection. IEEE Trans. Geosci. Remote Sens.; 2019; 58, pp. 1696-1717. [DOI: https://dx.doi.org/10.1109/TGRS.2019.2947708]

45. Seydi, S.T.; Amani, M.; Ghorbanian, A. A dual attention convolutional neural network for crop classification using time-series Sentinel-2 imagery. Remote Sens.; 2022; 14, 498. [DOI: https://dx.doi.org/10.3390/rs14030498]

46. Li, Z.; Chen, G.; Zhang, T. Temporal attention networks for multitemporal multisensor crop classification. Ieee Access; 2019; 7, pp. 134677-134690. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2939152]

47. Ramaswamy, H.G. Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision; Snowmass Village, CO, USA, 1–5 March 2020; pp. 983-991.