A Statistical and Efficiency-Driven Benchmark of Vision Transformer Architectures for Geospatial Intelligence Through Aerial Scene Classification

Land-use and land-cover (LULC) classification is crucial for interpreting high-resolution aerial and satellite imagery. This process supports many important activities, such as urban planning, disaster response, environmental monitoring, and geospatial intelligence [1-3]. Convolutional neural networks (CNNs) have long dominated visual recognition. However, the emergence of vision transformer (ViT) [4] demonstrated that transformer architectures based on self-attention can achieve competitive performance in large-scale image classification. This development has contributed to the broader adoption of transformer-based architectures for remote-sensing scene classification. Recent studies have further extended transformer-based remote-sensing classification toward reliability, uncertainty estimation, and interpretability, indicating a broader shift from accuracy-only evaluation toward more comprehensive assessment of model behavior [5]. The growing diversity of transformer architectures has consequently increased the need for systematic and multidimensional benchmarking. Existing comparative evaluations within remote-sensing scene classification have often emphasized aggregate predictive performance [6, 7], while less consistent attention has been given to statistical validation, computational profiling, quantitative feature-space analysis, and model interpretability [8, 9]. These considerations are particularly important for operational deployment, where computational requirements and inference efficiency can directly influence practical feasibility. Moreover, small differences in classification accuracy do not necessarily indicate statistically meaningful differences in predictive performance or practical deployment suitability.

To address this gap, this study conducts a comprehensive benchmark of five representative transformer-based architectures on the Aerial Image Dataset (AID). The objective is to evaluate predictive performance, alongside paired statistical validation, computational profiling, quantitative feature-space analysis, and gradient-based interpretability under a common evaluation framework. A supplementary evaluation under a reduced-training-data setting was also conducted to assess the sensitivity to training-data proportion of the comparative findings to training-data proportion. The five models were selected to represent three complementary dimensions of architectural and pretraining variation: (i) attention architecture, represented by vision transformer with 16 × 16 image patches (ViT-B/16) [4] and Swin Transformer V2-Base (Swin-V2-B) [10] (ii) pretraining paradigm, encompassing supervised ImageNet pretraining, self-supervised representation learning with DINOv2 ViT-B/14 [11], and masked image modeling (MIM) with knowledge distillation using EVA-02 ViT-B/14 [12] and (iii) pretraining domain, contrasting general-purpose ImageNet pretraining with remote-sensing-specific pretraining through Rotated Varied-Size Window Attention (RVSA-B), pretrained on MillionAID [13].

The contributions of this study are:

• A controlled comparison of five ViT architectures using a common fine-tuning protocol on the AID dataset.
• A rigorous statistical evaluation using McNemar's and Wilcoxon signed-rank tests with Holm-Bonferroni correction to distinguish statistically supported differences from small numerical variations.
• A computational efficiency analysis combining theoretical FLOPs with empirically measured inference latency to examine practical inference cost.
• A combined representation and interpretability analysis using feature-space embeddings, class-separability measures, and SmoothGrad to characterize learned representations and visual evidence underlying model predictions.

The remainder of this paper is organized as follows. Section 2 reviews related work in CNN- and transformer-based scene classification. Section 3 describes the dataset, model architectures, training protocol, and statistical and efficiency analysis methods. Section 4 presents the experimental results. Section 5 discusses their implications and limitations, and Section 6 concludes.

Figure 1 illustrates the overall workflow adopted in this study. The AID dataset [27] was partitioned into training, validation, and test subsets using a 70%/10%/20% split. Five ViT architectures, ViT-B/16, Swin-V2-B, DINOv2, EVA-02, and RVSA, were then fine-tuned under a common training and evaluation protocol to minimize variation arising from differences in experimental settings.

Following training, model performance was evaluated on the held-out test set using accuracy, precision, recall, and F1-score. The evaluation comprised four complementary analyses: (i) statistical analysis using bootstrap confidence intervals, McNemar's test, and Wilcoxon signed-rank test with multiple-comparison correction. (ii) computational profiling using model parameters, FLOPs, inference latency, GPU memory consumption, and Pareto analysis. (iii) feature representation analysis using t-distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP) together by quantitative measures of class separability. (iv) gradient-based interpretability analysis using SmoothGrad saliency maps.

tu_pian_14.png

Figure 1. Workflow of the comparative benchmarking framework

To assess the sensitivity of the comparative findings to the amount of labeled training data and to enhance comparability with prior AID studies commonly using a 20% training-data setting, an additional experiment was conducted using 20% of the dataset for training, 10% for validation, and the remaining 70% for independent testing. The validation subset was kept separate from the test set to ensure that early stopping and checkpoint selection were performed without access to test data. The supplementary experiment followed the same model architectures, preprocessing configurations, and classification metrics as the primary benchmark.

3.1 Dataset description

Empirical evaluations were executed on the AID benchmark [27], which contains 10,000 RGB images spanning 30 distinct LULC scene classes. The images are 600 × 600 pixels and were collected from multiple remote-sensing sources with spatial resolutions ranging approximately from 0.5 to 8.0 m per pixel. The number of images per class ranges from approximately 220 to 420.

For the primary benchmark, a fixed stratified random split was used to create training, validation, and test subsets containing 70%, 10%, and 20% of the dataset, respectively. Stratification was used to preserve the class distribution across the subsets, and the three subsets were mutually exclusive. The split configuration is summarized in Table 1. The class-wise distributions are shown in Figure 2, and representative images from all 30 classes are presented in Figure 3.

The five transformer backbones were selected to represent different attention mechanisms and pretraining paradigms. Their principal architectural and pretraining characteristics are summarized in Table 2.

tu_pian_15.png

Figure 2. Class-wise image distribution across the training, validation, and test subsets

tu_pian_16.png

Figure 3. Representative images from the 30 scene classes from Aerial Image Dataset (AID)

Table 1. Aerial Image Dataset (AID) split summary

The models retain the input resolutions specified by their respective pretrained configurations. Swin-V2-B uses a 256 × 256 input, whereas ViT-B/16, DINOv2, EVA-02, and RVSA use 224 × 224 inputs. This difference is retained rather than artificially modifying the pretrained configurations. Consequently, input resolution is recognized as an experimental factor when interpreting differences in computational cost and predictive performance.

ViT-B/16 employs global multi-head self-attention to a sequence of non-overlapping 16 × 16 image patches. The backbone uses supervised pretraining on ImageNet-21K followed by fine-tuning on ImageNet-1K [4]. Swin-V2-B uses a hierarchical representation with shifted-window self-attention and was pretrained on ImageNet-21K [10]. Unlike global self-attention, window-based attention restricts computation to local regions, while shifted windows facilitate information exchange across neighboring windows. DINOv2 (ViT-B/14) represents the self-supervised pretraining paradigm. It employs self-distillation and was pretrained on the LVD-142M dataset, which contains approximately 142 million images [11]. The pretrained representation was transferred directly to the AID classification task without intermediate supervised ImageNet fine-tuning. EVA-02 (ViT-B/14) represents the MIM with feature-level knowledge distillation from a CLIP-derived vision teacher [12]. The evaluated backbone was pretrained using ImageNet-21K and transferred to the downstream AID classification task without intermediate supervised ImageNet fine-tuning. RVSA ViT-B represents the remote-sensing-specific pretraining paradigm. It employs Rotated Varied-Size Window Attention together with periodic global attention to model spatial patterns at different scales and orientations [13]. The backbone was pretrained using masked autoencoding on the MillionAID dataset [28]. For downstream AID classification, the pretrained classification head of each backbone was replaced with a randomly initialized fully connected layer containing 30 output units. All backbone parameters were subsequently fine-tuned.

All evaluated models build on the scaled dot-product self-attention mechanism. Given an input sequence of N tokens mapped to query (Q), key (K), and value (V) projections, the attention matrix is computed according to Eq. (1).

$Attention\left( Q,\text{ }\!\!~\!\!\text{ }K,\text{ }\!\!~\!\!\text{ }V \right)=softmax\left( \frac{Q{{K}^{T}}}{\surd d} \right)V$         (1)

where, d denotes the dimensionality of the key vectors.

The five backbones differ primarily in their attention organization and pretraining strategy, as summarized in Table 2.

Table 2. Structural and pretraining specifications of evaluated vision transformer (ViT) backbones

3.2 Training protocol

All five transformer backbones were fine-tuned using a common optimization protocol to minimize variation arising from differences in downstream training settings. The AdamW optimizer was used with a base learning rate of $1\times {{10}^{-4}}$, a weight decay of 0.05, and a batch capacity of 32 samples. A 3-epoch linear warm-up was followed by cosine learning-rate decay over the remaining training steps. Regularization included label smoothing with $\varepsilon =0.1$ and gradient-norm clipping at 1.0. Automatic mixed precision was used during training. Training was performed for a maximum of 30 epochs with early stopping based on validation accuracy and a patience of 15 epochs. The checkpoint achieving the highest validation accuracy was retained for subsequent evaluation. The held-out test set was not used for model selection or early stopping.

Label smoothing was incorporated into the cross-entropy objective. For an input with true class c, the regularized target vector ${{\tilde{y}}_{k}}\text{ }\!\!~\!\!\text{ }$is computed using Eq. (2).

$\tilde{y}_k=\left\{\begin{array}{c}1-\varepsilon+\frac{\varepsilon}{C}, \quad k=c \\ \frac{\varepsilon}{C}, \quad k \neq c\end{array}\right.$         (2)

where, $\varepsilon $ denotes thesmoothing factor and the total number of scene classes C is 30.

The corresponding cross-entropy loss is computed using Eq. (3).

$\mathcal{L}=-\sum_{k=1}^C \tilde{y}_k \log \left(p_k\right)$         (3)

where, ${{p}_{k}}$ denotes the predicted softmax probability for class k. The optimization step follows a segmented temporal learning rate function $\eta \left( t \right)$ as defined in Eq. (4).

$\eta(t)=\left\{\begin{array}{r}\eta_{\max } \frac{t}{t_w}, \quad t \leq t_w \\ \eta_{\max } \cdot \frac{1}{2}\left[1+\cos \left(\pi \frac{t-t_w}{t_{\max }-t_w}\right)\right],\end{array} \quad t>t_w\right.$         (4)

where, ${{\eta }_{max}}$ denotes the base learning rate, ${{t}_{w}}$ denotes the terminal step of the 3-epoch warmup, and ${{t}_{max}}$ denotes the total number of optimizer steps across a 30-epoch training horizon, and t denotes the current optimizer step. With a batch size of 32, t_w corresponds to the number of mini-batch updates performed during the first three epochs.

3.3 Evaluation metrics and statistical analysis

Classification performance was evaluated using overall accuracy (OA) together with per-class precision, recall, and F1-score. OA was computed using Eq. (5).

$O A=\frac{1}{N} \sum_{i=1}^N I\left(y_i=\hat{y}_i\right)$         (5)

where, N denotes the number of test images, ${{y}_{i}}$ the ground-truth label, ${{\hat{y}}_{i}}$ the predicted label and $\mathrm{I}(\cdot)$ is the indicator function.

For each class c, precision, recall, and F1-score were computed using Eqs. (6)-(8) respectively.

$Precisio{{n}_{c}}=\frac{T{{P}_{c}}}{T{{P}_{c}}+F{{P}_{c}}}$          (6)

$Recal{{l}_{c}}=\frac{T{{P}_{c}}}{T{{P}_{c}}+F{{N}_{c}}}$          (7)

$F{{1}_{c}}=\frac{2Precisio{{n}_{c}}Recal{{l}_{c}}}{Precisio{{n}_{c}}+Recal{{l}_{c}}}$     (8)

where, $T{{P}_{c}},\text{ }\!\!~\!\!\text{ }F{{P}_{c}},\text{ }\!\!~\!\!\text{ and }\!\!~\!\!\text{ }F{{N}_{c}}$ denote the true-positive, false-positive, and false-negative counts for class c, respectively. Macro-averaged metrics were obtained by averaging the class-wise values equally across all 30 scene categories, whereas weighted averages accounted for the number of test samples in each class.

To assess pairwise differences between models, all statistical analyses were conducted using predictions or class-wise performance values derived from the same fixed, held-out test split. Three complementary statistical procedures were employed. First, bootstrap resampling was used to estimate the uncertainty of each model’s test accuracy. The test set was resampled with replacement 10,000 times, and the resulting distribution of bootstrap accuracies was used to compute percentile-based 95% confidence intervals. Second, McNemar's test was applied to all ten pairwise model comparisons using paired image-level correctness. For each model pair, the numbers of images correctly classified by model A but incorrectly classified by model B, and vice versa, were used to construct the paired 2 × 2 contingency table. When the total number of discordant observations was 25 or greater, the continuity-corrected McNemar chi-square statistic was used; when fewer than 25 discordant observations were present, the exact binomial version was applied. Let ${{n}_{10}}$ denote the number of test images correctly classified by model A but incorrectly classified by model B, and ${{n}_{01}}$ the reverse. For model pairs with at least 25 discordant observations, the continuity-corrected McNemar statistic was computed using Eq. (9).

${{\chi }^{2}}=\frac{{{(\left| {{n}_{10}}-{{n}_{01}} \right|-1)}^{2}}}{{{n}_{10}}+{{n}_{01}}}$       (9)

For model pairs with fewer than 25 discordant observations $({{n}_{10}}+{{n}_{01}}<25)$, the exact binomial version of McNemar's test was applied.

Third, pairwise Wilcoxon signed-rank tests were conducted using the classification accuracy of the 30 AID scene classes as the paired observations. The class-wise accuracy difference dc between models A and B was computed using Eq. (10).

${{d}_{c}}=Accuracy_{c}^{A}-Accuracy_{c}^{B}$         (10)

The corresponding Wilcoxon signed-rank statistic was computed using Eq. (11).

$W=\underset{c}{\mathop \sum }\,{{S}_{c}}{{R}_{c}}$           (11)

where, ${{R}_{c}}$ denotes the rank of $\left| {{d}_{c}} \right|$, and ${{S}_{c}}=sgn\left( {{d}_{c}} \right)$. Unlike McNemar's test, which evaluates paired image-level predictions, the Wilcoxon signed-rank test evaluates whether the class-wise performance differences are systematically shifted. Because ten pairwise model comparisons were conducted for each statistical test, Holm-Bonferroni correction was applied independently to both the McNemar and Wilcoxon test families to control the family-wise error rate at $\alpha =0.05$.

3.4 Computational efficiency analysis

Computational efficiency was evaluated using four complementary metrics that include total parameter count, FLOPs per image, inference latency, and peak GPU memory allocation during a single forward pass. FLOPs were estimated using Eq. (12).

$FLO{{P}_{S}}=2\times MA{{C}_{S}}$       (12)

The calculation follows the convention that one multiply-accumulate (MAC) operation consists of one multiplication and one addition and is therefore counted as two FLOPs.

Inference latency was measured on an NVIDIA RTX 3060 GPU using a batch size of one image. After ten warm-up iterations, inference time was averaged over 50 GPU-synchronized runs (R). Inference latency was computed using Eq. (13).

$Latency=\frac{1}{R}\underset{r=1}{\overset{R}{\mathop \sum }}\,{{t}_{r}},\text{ }\!\!~\!\!\text{ }r=1,\ldots ,R$        (13)

where, ${{t}_{r}}$ denotes the elapsed wall-clock inference time for run r. Reported latency values are presented as the mean $\pm $ standard deviation across the 50 runs. Peak GPU memory allocation was recorded during a single forward pass.

To characterize the trade-off between classification accuracy and computational cost, Pareto frontiers were constructed separately using FLOPs and measured inference latency as the cost metric. A model was considered Pareto-optimal if no other model achieved higher accuracy with equal or lower computational cost.

3.5 Interpretability analysis

To enable a consistent architecture-independent analysis, SmoothGrad [29], built upon vanilla input-gradient saliency [30], was employed. For an input image x and target-class score ${{f}_{c}}\left( x \right)$, vanilla saliency ${{M}_{c}}\left( x \right)$ is defined by Eq. (14).

${{M}_{c}}\left( x \right)=\frac{\partial {{f}_{c}}\left( x \right)}{\partial x}$         (14)

SmoothGrad was then computed by averaging the input gradients obtained from n = 20 Gaussian-noise perturbations, as defined in Eq. (15).

$\widehat{M}_c(x)=\frac{1}{n} \sum_{i=1}^n M_c\left(x+g_i\right), \quad g_i \sim N\left(0, \sigma^2 I\right)$    (15)

where, $\sigma =0.15$ denotes normalized pixel units. Averaging multiple noisy gradients reduces the visual noise commonly observed in single-gradient saliency maps while preserving the underlying discriminative regions. SmoothGrad does not directly expose internal attention weights; however, because it is based on input gradients, the same attribution procedure can be applied across all five architectures, providing a consistent basis for qualitative comparison.

The additional analyses provide several perspectives on the comparative behaviour of the five evaluated architectures. The results indicate that differences in training behaviour, computational cost, feature representation, and qualitative model responses are not completely captured by test accuracy alone.

DINOv2 exhibited a distinct training behaviour from the other evaluated models during preliminary experimentation. Under the initial early-stopping configuration, its validation accuracy was still improving when training was stopped. The study therefore implemented a common 15-epoch early-stopping patience for all five models, as described in Section 3.2. This observation motivated the use of a sufficiently long common patience rather than architecture-specific stopping criteria. However, it should not be interpreted as evidence that DINOv2 or self-supervised representations generally require slower convergence. Further experiments with architecture-specific learning schedules, learning rates, or backbone-freezing strategies would be required to examine this aspect more systematically.

The computational results showed that theoretical FLOPs and measured inference latency did not produce the same model ordering. Swin-V2-B had the lowest theoretical computational cost (30.48 GFLOPs), but its measured inference latency was higher than that of ViT-B/16, DINOv2, and EVA-02. Conversely, ViT-B/16 achieved the lowest measured latency in the benchmark. As reported in Table 7, theoretical FLOPs did not consistently correspond to measured inference latency across the evaluated architectures. This indicates that FLOPs alone do not fully characterize runtime under a specific hardware and software configuration. Therefore, measured inference latency provides a useful complementary efficiency measure, while the absolute latency values reported in this study should be interpreted as configuration-specific rather than as universal deployment estimates.

The feature-space analysis provides a complementary perspective on the classification results. EVA-02 obtained the highest silhouette coefficient (0.824) and inter/intra-class distance ratio (9.34). In contrast, Swin-V2-B achieved the highest test accuracy while ranking second on both feature-separation measures. ViT-B/16 achieved the second-highest test accuracy but obtained the lowest values for both feature-space measures. Thus, the ordering of the feature-space metrics did not directly correspond to the ordering of classification accuracy. These results suggest that pre-logit feature organization and final classification performance capture related but distinct aspects of model behaviour.

The qualitative saliency analysis in Figure 11 also showed differences in the image regions highlighted by the evaluated models for the selected correctly classified examples. Several models emphasized visually plausible scene structures, while some examples showed differences in the location and concentration of the highlighted regions. For example, the RVSA result for the Desert sample emphasized the boundary of the water body more strongly than the other models. Such differences indicate that models producing the same prediction may highlight different image regions for individual inputs. However, because the analysis was limited to six representative correctly classified examples, these observations remain illustrative rather than sufficient to establish systematic differences in model reasoning or attention behavior.

The supplementary reduced-training-data experiment provided an additional check on the effect of training-data proportion. As reported in Table 9, reducing the training proportion from 70% to 20% decreased test accuracy for all five models. Swin-V2-B remained the highest-performing model and DINOv2 remained the lowest-performing model, although RVSA and EVA-02 exchanged their relative positions. Because the supplementary and primary protocols used different held-out test sets, these results are interpreted descriptively rather than as a paired statistical comparison. The experiment therefore provides a supplementary indication of the consistency of the broad performance ordering under a reduced training-data setting, rather than a direct statistical validation of the primary results.

Taken together, the results show that the five architectures exhibit different characteristics when evaluated across predictive performance, statistical comparison, computational efficiency, feature-space organization, and qualitative saliency. Swin-V2-B achieved the highest test accuracy, but the pairwise analysis showed that only its comparisons with DINOv2 and EVA-02 were statistically significant after Holm-Bonferroni correction. ViT-B/16 provided the lowest measured latency in the original benchmark, while EVA-02 obtained the highest feature-separation measures. The supplementary reduced-training experiment preserved the highest and lowest positions in the accuracy ranking, although the middle-ranked models changed positions. These findings support the use of multiple evaluation criteria when comparing transformer architectures for aerial scene classification rather than relying on accuracy alone.

5.1 Limitations

Several limitations should be considered when interpreting the findings of this comparative evaluation.

First, the evaluation was conducted on a single benchmark dataset, AID. Although this study evaluated the five architectures across classification performance, statistical significance, computational efficiency, feature-space organization, and interpretability, evaluation on additional aerial scene classification datasets, such as NWPU-RESISC45 and UC Merced, would be useful for assessing whether the observed comparative patterns extend beyond the AID dataset.

Second, all five architectures were evaluated under a common training and hyperparameter protocol. This choice provided a consistent basis for comparison but did not represent the peak performance that could potentially be obtained through architecture-specific tuning. The models utilized the same optimizer, learning rate schedule, augmentation policy, batch size, and early-stopping patience. Therefore, the reported results should be interpreted as performance metrics under a unified evaluation protocol, rather than as the absolute maximum achievable performance of each architecture.

Third, each architecture was evaluated using a single training run. Training variability associated with stochastic optimization was therefore not systematically assessed. Multi-seed experiments would provide additional evidence regarding the stability of the observed performance differences. The bootstrap confidence intervals and paired statistical tests reported in this study provide complementary evidence based on fixed test-set predictions, but they do not assess variability across independent training runs.

Fourth, computational measurements were obtained using a single hardware and software configuration based on an NVIDIA RTX 3060 GPU and its corresponding PyTorch/CUDA environment. Consequently, inference latency and peak GPU memory allocation may differ on alternative hardware or software configurations. The reported efficiency measurements should therefore be interpreted within the evaluated computational environment rather than as universal deployment characteristics.

Finally, the primary experiment used a 70/10/20 train/validation/test split, whereas commonly reported AID evaluation protocols historically utilized different training proportions and validation arrangements. Consequently, the accuracy values reported here should not be directly compared with results obtained under substantially different split protocols. The 70/10/20 configuration was selected to provide a dedicated validation set for checkpoint selection and early stopping. To examine the sensitivity of the comparative results to training-data proportions, a supplementary 20/10/70 experiment was conducted and reported in Section 4.7. The highest- and lowest-ranked models remained unchanged, although EVA-02 and RVSA exchanged their relative positions, while absolute accuracy values decreased. Because the supplementary experiment used a different test set, it is treated as a sensitivity analysis based on aggregate accuracy rather than as a paired statistical comparison with the primary experiment.