H-EMFR: A Hybrid CNN–Transformer Framework for Lightweight Multi-Scale Small-Object Classification

The recognition of small objects remains a key and unresolved issue in computer vision due to the limited number of pixels available to provide spatial awareness, the loss of fine spatial information, and the impact of background clutter. Each of these constraints contributes to ambiguous features and loss of discriminability in deep models. Furthermore, real-world monitoring applications which may include UAV-based inspection, remote sensing, surveillance, and analytical use at the edge require accurate recognition performance, along with stringent computational constraints. Thus, the dual need for accuracy with efficiency presents a valuable research direction when developing lightweight efficient small-object classification frameworks.

Convolutional neural networks (CNNs) in a deep learning context can efficiently model local patterns [1]. CNN typically down sample progressively through the model captured cues at a small scale. Alternatively, transformer models [2] can efficiently capture long-range dependencies. However, they can be costly computationally which can limit their use for real-time applications including deployment on edge-based devices. Finding ways to bridge the trade-off noted above has inspired several lightweight hybrid approaches employing multi-scale convolutional-fusion models that utilize attention and transformer states. However, most models explored in the literature primarily rely on a detection approach to classification or they simply apply attention using the prior scale reasoning. Based on this gap the primary objective of this thesis is to develop a unified low-cost and hybrid framework for lightweight small-object classification.

Various recent investigations have tackled the challenge from complementary angles. A Lightweight Multi-Scale Network (LMSN) [3] that fuses receptive field enhancement and channel attention mechanisms. The LMSN construction illustrates the powerful properties of multi-scale fusions as a small-scale visual pattern detector. A similar, but different methodology is TinyDet [4], it is designed feature pyramids and prediction heads to be able to detect small-object detection with strict FLOP constraints, results demonstrated the efficacy of structural optimization over heavier detectors. A multi-dimensional trans-attention [5] streamlined detector for remote sensing images, TA-YOLO, using spatial and channel attention to help effectively detect small-target cues. Although these architectures presented substantial detections optimized for detection are not suitable for classification pipelines that evolve from labels and not bounding boxes.

Coincidentally, lightweight vision transformers have emerged, reshaping model designs for mobile and embedded deployment. The study MobileViT [6] used a hybrid CNN-transformer block to preserve convolutional efficiency, but added a transformer-level global context, improving the model’s usability for mobile vision reporting the model performed well on mobile vision. For example, a Hybrid CNN-Transformer Feature Fusion Network (HCT-FFN) [7] for single-image demonstrated that task-specific hybridization is still able to preserve fine level information with complicated distortion. A lightweight CNN-transformer architecture [8] for crop mapping using Sentinel-2 imagery effectively modelled both local textures as well as long-term spectral dependencies. Thirdly, many CNN-Transformer [9] hybrids generally favour accuracy over computational lightness. It avoids parametric pruning and low-rank attention or scale-adaptive fusion methods needed for inference in real-time on embedded devices. EdgeViT [10] similarly, indicated that edge deployable vision transformers could rival mobile CNNs with the experimental method of merging token mixing and localized attention, fundamentally changing the advantages of lightweight models.

Recent hybrid models have extended this idea to the domain of specific tasks with recent results. An ultra-lightweight convolution-transformer [11] for early fire-smoke detection, with a hybrid implementation of shallow transformers for higher responsiveness and depth wise convolutions to minimize overall resource use. In light of this development, revisited the scaling laws of vision transformers to reach the latency of MobileNets [12] without compromising accuracy, with a focus on structural re-parameterization and lightweight attention. Overall, these studies illustrate that hybridizing CNN–transformer architectures is a constructive avenue for matching efficiency and accuracy. Together these works support the utility of Hybrid CNN–Transformer architectures; however, they do not directly address small-object classification on satellite imagery, where the retention of pixel level chart and contextual coherence across recycling remains an unresolved question.

To address the mentioned gaps, this paper introduced Hybrid Efficient Multi-Scale Feature Refinement (H-EMFR). It is a new framework for lightweight small-object classification combines local convolutional locality, transformer-driven global reason, and adaptive aggregation from the four key module frameworks. The four modules are: a Lightweight Convolutional Extractor (LCE) for high-resolution texture encoding, an Attention-based Feature Refinement (AFR) block that aggregates both spatial and channel attention to improve discriminative cues, a Transformer-guided Context Aggregator (TGCA) that uses low-rank self-attention to use, and A Dynamic Multi-Scale Fusion (DMF) unit, that learns to adaptively re-weight scale specific features.

These components allow the model to capture fine spatial details without sacrificing efficiency applicable to edge latency. In addition, structured pruning and low-rank approximation to limit the parameter costs without loss of recognition accuracy. The model extensively tested on benchmark small-object datasets and results show outperformed recent lightweight networks and transformer-based paradigm baselines by 4-5%. It is on classification while reducing parameters by 40% supporting that adaptive hybrid is a viable strategy to balance other criteria.

The proposed model contributes the following:

• Designed a novel hybrid-lightweight architecture called H-EMFR, it incorporates convolutional and transformer-based feature refinement for small-object classification.

• Introduced a DMF, a strategy to adaptively balance local and global cues via learned scale weights.

• Incorporating a low-rank attention and structured pruning to maintain high efficiency on edge-device constraints.

• Offered an extensive evaluation showing state of the art accuracy efficiency trade-offs against front-runner of lightweight and attention-enhanced baselines.

For the rest of the paper, Section 2 reviews related work on lightweight models, small-object methods, and CNN–Transformer hybrids. Section 3 discusses proposed H-EMFR details and architecture. Section 4 describes datasets, experiment setup, metrics, and also reports both quantitative and qualitative results. And Section 5 concludes with limitations and future work.

This section describes the proposed H-EMFR architecture. The H-EMFR model has three main modules: Lightweight Convolutional Encoder (LCE), Adaptive Feature Refinement (AFR), TGCA, and DMF on a lightweight classification backbone. Figure 1 gives the architecture and data flow of a proposed H-EMRF framework. For an input image, the first step LCE is to perform feature extraction. This process is achieved via a lightweight convolution encoder. The LCE produces a hierarchy of feature maps $\left\{F_1, F_2, F_3\right\}$ of multi-scales which retain the relationships between fine and coarse resolutions. For example, the refiner module processes each multi-scale refined feature map using the AFR method. The AFR method takes the advantage of joint channel and spatial attention to increase the local discriminative cues in order to keep the fine and coarse details that are necessary for detecting small objects. It then outputs the refined features which are passed to the transfor-guided contextual aggregation module.

Figure 1. Hybrid Efficient Multi-Scale Feature Refinement (H-EMFR) framework design architecture

In the TGCA module, the refined features are combined to create similar tokens. Using low-rank self-attention the tgcamod can capture global context items with less computational overhead than a full-scale attention method. The TGCA takes the contextual representation, which gives both a scene-context level and cross-scale relational information, to complement the local features.

In the last problem, the DMF module dynamically adapts and combines the refined multi-scale features based on the individually learned importance coefficients for each trained model's sample. This produces a unified high-resolution feature representation which then forms part of the input for the classification head. The ordered flow of processing through the pipeline first creates local refinement from multiple sources and then aggregates all local refinements using long-distance contextual information and creates an adaptive fusion of all features generated by the training dataset to create new features from the context provided by the long-distance connection. Thus, fine spatial resolutions can be maintained while efficiently and effectively integrating geographically disparate and highly correlated sources of information.

The first design principle is the importance of ordering the AFR module, which is applied prior to the information being aggregated across the entire image in the TGCA, retaining detailed information about fine-level local cues before they are incorporated across spatial positions in the information algorithm through global attention in the TGCA. This design also ensures that high-frequency small-object features are retained and not lost during the global attention mechanism. Secondly, the low-overhead attention seen in TGCA uses low-rank or linearized self-attention mechanisms in which much of the computational complexity and memory usage overhead is a result of attention-based information. This design can provide attention capabilities and long-range dependencies, but not at the computational cost of more complex methods.

Third, adaptive fusion is accomplished using the DMF unit learns per-sample scale-dependent weights that allow the network to adaptively favour higher-resolution feature maps for small objects and coarser representations for larger objects. Finally, computation-aware compression techniques will be incorporated to reduce inference cost while preserving accuracy. Collectively, these principles form H-EMFR to be an efficient, scalable, and contextually aware model designed for small object visual recognition in resource-limited settings.

3.1 Lightweight Convolutional Encoder

The LCE acts as the core feature extractor for H-EMFR with the goal of reducing computation while efficiently learning hierarchical texture and spatial patterns from the input image. Given an input image $I \in R^{3 \times H_0 \times W_0}$, the encoder will output three feature maps that have a spatial resolution that is progressively smaller and the representation is increasingly semantic as shown in Eq. (1).

$F_i=\operatorname{LCE}_i(I), i \in\{1,2,3\}$              (1)

Here, $F_i \in R^{C_i \times H_i \times W_i}$ denotes the feature tensor at the $\mathrm{i}^{\text {th}}$ scale, $C_i$ is the channels with spatial dimensions of $H_i \times W_i$ such that $H_1>H_2>H_3$. Each of the hierarchical representations, $F_1, F_2$, and $F_3$ represent high-resolution shallow, mid-level, and low-resolution deep representations respectively, thus allowing a network to learn both finegrained object and semantic object cues. To reduce computations, every convolutional block of LCE uses depth wise separable convolutions and inverted residual bottlenecks inspired by MobileNetV3 and ghost operations to minimize redundant computation of features.

A standard convolution operation using $C_{\text {input}}$ and output channels with kernel size $k \times k$ has an $O\left(C^2 k^2\right)$ complexity. H-EMFR employs depthwise separable convolutions to improve efficiency by decomposing the standard convolution operation into two sequential operations.

$\operatorname{Conv}_{d w}(x)=\operatorname{Conv}_{p w}($DepthwiseConv $(x))$                (2)

Here, x is the input feature map, DepthwiseConv $(x)$ is the depth wise convolution applies each kernel separately to each of the input channels, and $\operatorname{Conv}_{p w}$ is the pointwise convolution ($1 \times 1$) linearly combines the output from the depth wise convolution across all channels. As a consequence, the computational cost reduces to $O\left(C k^2+C^2\right)$ thereby delivering considerable improvement in efficiency without any appreciable loss in representational ability.

Each of the LCE stages uses an Inverted Residual Block which efficiently propagates features while remaining parameters efficient. The structure of the Inverted Residual Block is as given as Eq. (3).

$\operatorname{IRB}(x)=\operatorname{Conv}_{1 \times 1}\left(\operatorname{Conv}_{d w}\left(\operatorname{Conv}_{1 \times 1}^{\text {expand}}(x)\right)\right)+x$                   (3)

Here, $\operatorname{Conv}_{1 \times 1}$ denotes expansion layer expands the channel dimensionality allowing richer spatial encoding to be utilized. The depthwise convolution then applies spatial filtering, while the projection convolution decreases the dimensionality back to the input channel. The residual shortcut connection maintains gradient flow, and stabilizes training, which is especially helpful in low-capacity models.

3.2 Adaptive Feature Refinement

The AFR module acts on each scale feature map to provide local discriminative cues before global mixing occurs. This local refinement is critical for detecting small-objects as small-region activations will not be weighted too low. The AFR achieves this by utilizing lightweight spatial attention and efficient channel attention (ECA) and combining those ideas in a residual gating scheme. Given an input feature map $R^{C \times H \times W}$, Eq. (4) begin a channel squeeze using global average pooling.

$\mathrm{z}_{\mathrm{c}}=\frac{1}{\mathrm{HW}} \sum_{\mathrm{h}=1}^{\mathrm{H}} \sum_{\mathrm{w}=1}^{\mathrm{W}} \mathrm{F}_{\mathrm{c}, \mathrm{h}, \mathrm{w}}$                (4)

Here, $z_c \in R^C$ is the global response of each channel, $F_{c, h, w}$ is the activation value of the feature map at channel c , height h , and width $\mathrm{w}, \mathrm{H}$ is the height of the feature map, W is the width of the feature map. Model next perform channel reweighting using global average pooled descriptors, with a 1 D convolution with the convolutional kernel weights applicable across channel descriptors. Thus, the Efficient Channel Attention (ECA) produces adaptive weights as given in Eq. (5).

$w_c=\sigma\left(\operatorname{ConvlD}_k\left(z_c\right)\right)$              (5)

Here, k is the kernel size controlling the local cross-channel dependency and σ(⋅) is the sigmoid activation. The weights are applied to adaptively reweight the importance of each channel by emphasizing important filters. Thus, we can reweight the input feature by learned attention coefficients as follows and acquire the focused channel-attended feature as given in Eq. (6).

$F_{c a}=w_c \odot F$               (6)

Here, F is the input feature map tensor of shape $C X H X W$, $w_c$ is the channel attention weight vector, and $\odot$ is the element-wise multiplication with broadcasting on the spatial dimensions. Additionally, to improve the localization further we compute spatial attention by aggregating channel information using Eq. (7).

$\mathrm{F}_{\mathrm{avg}}=\operatorname{AvgPool}_{\mathrm{ch}}(\mathrm{F}), \mathrm{F}_{\max }=\operatorname{MaxPool}_{\mathrm{ch}}(\mathrm{F})$              (7)

Here, $F_{\text {avg }} \in R^{1 \times H \times W}$ is the average of the activation responses for all channels at each spatial point providing information about how active the features are in-spatial locations. $F_{\text {max }} \in R^{1 \times H \times W}$ the strongest channel response at each location captures and emphasizes the most locally salient features or edges. It captures general intensity or global consistency of activation across features.

This provides a way for the model to jointly reason about both the average activations and the max activations when making inference about spatial importance. Both of these salient maps are encodings of complementary information. $F_{\text {avg }}$ and $F_{\max}$ are concatenated along the channel dimension to form a spatial descriptor of the original information as given in Eq. (8).

$\left[\mathrm{F}_{\text {avg}} ; \mathrm{F}_{\max }\right]=R^{2 \times \mathrm{H} \times \mathrm{W}}$             (8)

Here, $F_{a v g}$ encodes distributed encodings such as background and contextual based patterns, and $F_{\max }$ codes localized salient features or edges as high-contrast information. A $3 \times 3$ convolutional layer is then applied to this concatenated map as given in Eq. (9).

$\mathrm{M}_{\mathrm{s}}=\sigma\left(\operatorname{Conv}_{3 \times 3}\left(\left[\mathrm{~F}_{\mathrm{avg}} ; \mathrm{F}_{\mathrm{max}}\right]\right)\right)$            (9)

Here, $\operatorname{Conv}_{3 \times 3}(\cdot)$ applies local filtering operations to learn a spatial attention mask weights from local neighbouring relationships across a $3 \times 3$ window, $\sigma(\cdot)$ is the sigmoidal activation function which normalizes a value into the range to $[0,1]$ is essentially a spatial attention map $M_s \in[0,1]^{H \times W}$.

The mask $M_s$ acts like a soft gate by emphasizing the important regions (values close to 1 ) while reducing irrelevant, or noisy parts (values near 0 ). In the next refinement stage, we apply this mask to the feature tensor (from channel attention) element-wise as given in Eq. (10).

$F_{s a}=M_s \odot F_{c a}, F_{\mathrm{ref}}=F+\beta F_{s a}$               (10)

Here, β is a learnable scalar that balances the refinement strength, and ⊙ represents element-wise multiplication with broadcasting across channels.

3.3 Transformer-Guided Context Aggregation

TGCA utilizes global context and cross-scale relational cues to disambiguate localized small-object activations with context support. To retain computational efficiency, we utilize a low-rank self-attention leveraging approximations inspired by Linformer-style architecture. Tokens are formed by the refined feature maps using the following approach. Applied spatial adaptive pooling to each $F_{\text {refi}}$ producing $T_i$ tokens for each scale. It is a light-weight mechanism.

This creates tokens $T=\left[T_1 ; T_2 ; T_3\right]$ resulting in a sequence of input for $X \in R^{N X d}$. At each resolution, refined features are pooled down to a predetermined number of tokens. Each refined feature map $F_{\text {ref}_i}$ is tokenized via adaptive pooling as given in Eq. (11).

$\begin{gathered}T_i=\operatorname{PoolTokens}\left(F_{\text {ref}_i}\right), X=\operatorname{Concat}\left(T_1, T_2, T_3\right) \in \mathbb{R}^{N \times d}\end{gathered}$              (11)

Here, N is the quantity of total tokens and d is the dimension of the embedding, $F_{\text {ref} i}$ is a feature map, $T_i$ is the produced token. Query, key, and value projections are computed using Eq. (12).

$Q=X W_Q, K=X W_K, V=X W_V$              (12)

Here, X is the input feature matrix, $W_Q, W_K, W_V \in R^{d \times d_k}$ are learnable linear projection matrices, Q is the query matrix, K is the key matrix, and V is the value matrix. To enable attention with reduced quadratic complexity, low-rank projections $P_K, P_V \in R^{(r \times N)(r \ll N)}$ are utilized to compress the key and value matrices using Eq. (13).

$K^{\prime}=P_K K, V^{\prime}=P_V V$                (13)

Here, $P_K$ is the projection transformation matrix applied to keys, $P_V$ is the projection transformation matrix applied to values, and $K^{\prime}, V^{\prime}$ are refined value and key representation. It reducing FLOPs and memory demand without compromising global context. The attention scores and normalized attention weights are computed using Eq. (14).

$S=\frac{Q\left(K^{\prime}\right)^{\top}}{\sqrt{d_k}}, A=\operatorname{softmax}(S)$               (14)

Here, Q is the query matrix derived from the input matrix, $K^{\prime}$ is denoted as refined key matrix, $S \in R^{N \times r}$ captures token-to-key similarity. The weighted aggregations of attended features and multiple attention heads are computed using Eqs. (15) and (16).

$Y=A V^{\prime}$              (15)

$M H(X)=\operatorname{Concat}\left(Y_1, \ldots, Y_h\right) W_O$           (16)

Here, h is the number of attention heads, X is the feature tensor, $W_O$ is the linear projection weight matrix, and A is attention map computed. Each TGCA block ends with a second normalization and feed-forward refinement using Eqs. (17) and (18).

$\operatorname{FFN}(Y)=W_2 \operatorname{RELU}\left(W_1 Y+b_1\right)+b_2$           (17)

$Y_{\text {out }}=\mathrm{LN}(X+\mathrm{MH}(X))+\mathrm{LN}(\mathrm{FFN}(Y))$            (18)

Here, Y is the output from multi head attention, $W_1, W_2$ are learnable weight matrices, $b_1, b_2$ are the biases, and $\operatorname{GELU}()$ is the activation function. This class of structure yields compact tokens that can integrate global contextual reasoning. The tokens with context are pooled via global readout pooling to reduce multiple tokens with context into one single representation.

$F_{\mathrm{ctx}}=R_{\text {out}}\left(Y_{\text {out}}\right)$             (19)

Here, $F_{\mathrm{ctx}}$ is the context aware global feature vector, $R_{\text {out}}(.)$ global summarization function, and $Y_{\text {out}}$ is the refined normalized token feature. This results in a global feature to provide an object-scale relation summary.

3.4 Dynamic Multi-Scale Fusion

DMF adaptively re-weights the refined multi-scale features such that the importance of an individual sample. A specific region is preserved the network should learn the importance of $F_{\text {ref1}}$ (high-res) for small objects while $F_{\text {ref2}}$ (low-res) may have higher importance along with its mid-res counterparts for larger objects. Each scale represents a portion of the overall importance using Eqs. (20) and (21).

$s=\operatorname{MLP}\left(\operatorname{GAP}\left(\left[\operatorname{Pool}\left(F_{\mathrm{ref}_1}\right) ; \operatorname{Pool}\left(F_{\mathrm{ref}_2}\right) ; \operatorname{Pool}\left(F_{\mathrm{ref}_3}\right) ; F_{\mathrm{ctx}}\right]\right)\right)$               (20)

$\alpha=\operatorname{softmax}(s), \quad \sum_i \alpha_i=1$               (21)

Here, $\alpha_i$ is the implied importance weight for scale $\mathrm{i}, \,F_{\text {ref}_i}$ is the refined convolutional feature, $\operatorname{Pool}($.$)$ is the spatial pooling operator, GAP(.) is the Global Average Pooling, and MLP(.) is the Lightweight multi-layer perceptron. The resulting fused representation is achieved by up sampling all features to a specific resolution which allows the weighted aggregation is calculated using Eq. (22).

$F_{\text {fused }}=\sum_{i=1}^3 \alpha_i \cdot \operatorname{Upsample}\left(F_{\mathrm{ref}_i}, H_f, W_f\right)$             (22)

Here, $H_f, W_f$ are target height and width for fusion, $F_{f u s e d}$ is the final multi scale fused feature map, and $\alpha_i$ is an adaptive weight for each scale yielding a single high-resolution representation.

A global average pool (GAP) should be performed above the fused feature, followed by a small FC bottleneck where softmax cross-entropy is applied. A global descriptor v and class logits will be calculated using Eq. (23).

$v=\operatorname{GAP}\left(F_{\text {fused}}\right)$               (23)

It is projected into class logits via a fully connected layeras shown in Eq. (24).

logits $=W_c v+b_c W_c \in \mathbb{R}^{C_{\text {out}} \times d_v}$            (24)

Here, v is the final layer fused feature vector, $W_c$ is the classification weight matrix, $C_{\text {out}}$ is the number of output classes, $d_v$ feature dimension of v , and $b_c$ bias term of each output. The mapped feature represents class probabilities. The learning objective combines the classification losses and distillation losses. The model is mainly trained with crossentropy loss using Eq. (25).

$\mathcal{L}_{C E}=-\sum_{c=1}^{C_{\text {out}}} y_c \log \left(\operatorname{softmax}(\operatorname{logits})_c\right)$             (25)

and temperature-scaled KD using Eq. (26).

$\mathcal{L}_{K D}=T^2 D_{K L}\left(\operatorname{softmax}\left(\frac{Z_s}{T}\right) \| \operatorname{softmax}\left(\frac{Z_t}{T}\right)\right)$               (26)

Here, T is the temperature parameter, and $z_s, z_t$ are logits from student and teacher network. The final training loss combines both terms with a regularization penalty using Eq. (27).

$\mathcal{L}_{\text {total }}=\mathcal{L}_{C E}+\lambda_1 \mathcal{L}_{K D}+\lambda_2\|\theta\|_2^2$           (27)

Here, $\lambda_1$ and $\lambda_2$ adjust the trade-off between distillation and weight decay, $\mathcal{L}_{K D}$ is the KD loss, and $\theta$ is the set of trainable parameters. Additional auxiliary losses such as the AFR contrastive sharpener and pruning aware L1 regularization can optionally augment small-object sensitivity and structured sparsity.

In summary, the H-EMFR framework provides a hierarchical and computation aware small-object recognition pipeline, where the LCE efficiently encodes multi-scale features, the AFR sharpening local cues, the TGCA captures long-range dependencies through low-rank attention and the DMF adaptively fuses these representations as a function of the object scale. A balanced end-to-end formulation provides the necessary accuracy, efficiency and scaling design for edge-aware visual recognition tasks.