An Efficient Hybrid Vision Transformer for Topology-Preserving Retinal Vessel Segmentation

Retinal vessel segmentation has become an essential task in ophthalmic image analysis because the retinal vascular network provides valuable anatomical and physiological biomarkers for the diagnosis, monitoring, and prognosis of numerous ocular and systemic diseases, including diabetic retinopathy, glaucoma, hypertensive retinopathy, and cardiovascular disorders [1-3]. Morphological characteristics such as vessel calibre, tortuosity, branching geometry, vascular density, and vessel connectivity reflect pathological alterations in retinal circulation and therefore play a critical role in computer-aided clinical decision-making. Automated vessel segmentation in retinal fundus imaging is gaining attention for its non-invasive nature, low cost, and broad accessibility, making it an effective tool for large-scale ophthalmic screening. Nevertheless, accurate segmentation remains challenging because retinal images exhibit substantial variations in illumination, imaging artefacts, low vessel-to-background contrast, pathological abnormalities, and highly heterogeneous vessel diameters. Furthermore, the presence of thin capillary branches and intricate bifurcation structures significantly increases segmentation difficulty, frequently leading to discontinuous vessel extraction, inaccurate vessel boundaries, and poor preservation of retinal vascular topology [4, 5].

Recent developments in deep learning have substantially improved retinal vessel segmentation performance. Convolutional Neural Networks (CNNs), particularly encoder–decoder architectures derived from U-Net, have demonstrated remarkable competency in learning hierarchical vessel representations through convolutional feature extraction and multi-scale information aggregation. Numerous attention-enhanced CNN models have further improved vessel boundary delineation and peripheral vessel detection by incorporating skip connections and adaptive feature refinement. Despite these advances, CNN-based methods have limited receptive fields, hindering their capacity to capture long-range spatial relationships and global vascular connectivity in the retinal network. To address these limitations, Vision Transformer (ViT) architectures have recently been introduced to exploit self-attention mechanisms for modelling global contextual dependencies and vessel topology. Transformer-based models have shown superior capability in preserving vessel continuity and structural consistency; however, they generally require large-scale annotated datasets, introduce considerable computational complexity, and often exhibit reduced sensitivity to extremely thin vessels due to patch-based feature representation. Consequently, neither CNNs nor Transformers alone provide an optimal balance between local vessel discrimination, global contextual modelling, and computational efficiency.

Although CNN-based encoder-decoder networks (e.g., U-Net and its variants) excel at extracting local vessel morphology, their limited receptive fields prevent them from modelling long-range vessel connectivity; this frequently results in fragmented thin vessels and broken bifurcation structures. Conversely, pure ViT architectures capture global context effectively but suffer from three practical drawbacks: (i) high computational cost (often exceeding 10 GFLOPs), (ii) reduced sensitivity to capillaries because of fixed patch tokenisation, and (iii) the need for large annotated datasets. Hybrid CNN–Transformer models partially mitigate these issues, yet most still rely on heavy feature-fusion blocks or deep Transformer backbones that compromise real-time applicability. Consequently, there remains a clear need for a lightweight architecture that simultaneously (a) preserves global vascular topology, (b) accurately localises fine vessels, and (c) maintains low parameter count and fast inference.

This paper proposes a novel Vessel-Aware Hybrid Transformer Bidirectional Segmentation Network (VAHT-BiSRNet) for automated retinal vessel segmentation. The framework integrates a Multi-Scale Vision Transformer (MSVT) encoder to capture long-range vascular dependencies, a Hybrid Feature Fusion Network (HFF-Net) to learn multi-scale vessel representations, and a Bidirectional Segmentation Refinement Network (BiSRNet) to facilitate vessel-aware feature interaction between global and local representations in Table 1. Furthermore, a hybrid attention mechanism combining Squeeze-and-Excitation (SE) with the Convolutional Block Attention Module (CBAM), hereafter referred to as Squeeze-and-Excitation Convolutional Block Attention Module (SE-CBAM), is employed to enhance vessel localisation and suppress background interference. Extensive experiments conducted on the DRIVE, CHASE_DB1, and STARE datasets demonstrate the effectiveness and robustness of the proposed framework.

The main contributions of this work are summarised as follows:

•A novel VAHT-BiSRNet architecture is introduced for retinal vessel segmentation by integrating CNN-based local learning with transformer-based global topology modelling.

•A MSVT encoder is developed to capture long-range vessel dependencies and maintain vascular continuity across multiple spatial resolutions.

•A HFF-Net and BiSRNet are presented to enhance multi-scale vessel representation and bidirectional feature interaction.

•A hybrid SE-CBAM attention mechanism is implemented to improve vessel localisation, reduce background noise, and preserve fine vascular structures, while the overall model remains lightweight (2.21 M parameters).

The remainder of this manuscript is organised as follows. Related works are presented in Section 2. The datasets, preprocessing procedures and the proposed hybrid CNN-ViT architecture are explained in Section 3. The experimental setup and performance assessment results are elaborated in Section 4. The work is concluded in Section 5 with important findings and suggestions for further research.

Table 1. Glossary of key modules and terms

3.1 Overview of the proposed Vessel-Aware Hybrid Transformer Bidirectional Segmentation Network framework

The proposed VAHT-BiSRNet is designed to accurately segment retinal blood vessels while preserving vascular continuity, bifurcation structures, and thin vessel details. Unlike traditional CNN-based architectures that mostly concentrate on local feature extraction, the proposed technique combines attention-guided refinement, vessel-aware feature fusion, and transformer-based global contextual learning. The complete architecture consists of seven major stages: image acquisition, preprocessing, MSVT encoding, HFF-Net, BiSRNet, SE-CBAM attention enhancement, and vessel mask decoding, shown in Figure 1. The whole architecture is especially built to preserve computational efficiency while capturing both global vessel structure and local vascular anatomy.

1.png

Figure 1. Overall architecture of the proposed Vessel-Aware Hybrid Transformer Bidirectional Segmentation Network (VAHT-BiSRNet) framework for retinal vessel segmentation

3.2 Datasets and preprocessing

The proposed framework was evaluated on three publicly available benchmark datasets: DRIVE, CHASE_DB1 and STARE. These datasets were deliberately selected because they present complementary challenges that comprehensively test the robustness of a vessel segmentation model.

DRIVE contains 40 colour fundus images (565 × 584) acquired from a diabetic retinopathy screening programme in the Netherlands. It is the most widely used benchmark and provides standardised annotations, making it ideal for controlled comparison. CHASE_DB1 comprises 28 images (999 × 960) captured from multi-ethnic school children; it exhibits larger vessel-width variation and higher inter-subject anatomical variability. STARE consists of 20 images (700 × 605) that include both healthy and pathological retinas with significant illumination variation and lesion interference. Together, the three datasets cover a broad spectrum of imaging conditions, vessel morphologies, and pathological presentations.

For all three datasets, a consistent 70: 15: 15 split (training: validation: testing) was adopted while preserving the original vessel-to-background pixel distribution in each subset. The split ratio was chosen to maximise training data while still providing statistically meaningful validation and test sets given the modest total image counts. Data augmentation (horizontal/vertical flips, rotations of ±25°, brightness/contrast jitter and random cropping) was applied exclusively to the training subset to prevent information leakage. Class imbalance is severe in all three datasets (vessel pixels typically constitute only 8–12% of the image); the composite Binary Cross-Entropy (BCE) + Dice loss (Section 3.7) was therefore employed to mitigate this imbalance.

Prior to feature extraction, retinal images undergo a comprehensive preprocessing pipeline to improve vessel visibility and reduce image variability. Initially, image quality assessment is performed to identify blurred, underexposed, or low-contrast images that may adversely affect segmentation performance. To ensure uniform intensity distributions across the datasets, all the images are now normalized and scaled to a uniform resolution. The Normalized standard image size is $\mathrm{X} \in R^{512 \times 512 \times 3}$ intensity distributions and improves robustness of vessel feature extraction across the given datasets, as mentioned in Eq. (1). Images with low $V_{\text {blur}}$ values are considered as blurred and may negatively affect vessel segmentation performance is shown in Eq. (2).

$I_{\text {norm}}=\frac{I-I_{\min}}{I_{\max}-I_{\min}}$       (1)

where, $I=$ original retinal image, $I_{\min}=$ minimum pixel intensity, $I_{\text {max}}=$ maximum pixel intensity, $I_{\text {norm}}=$ normalized image.

$V_{\text {blur}}=\operatorname{Var}\left(\nabla^2 I\right)$       (2)

where, $\nabla^2 I=$ Laplacian operator, $\operatorname{Var}(\cdot)=$ variance function, $V_{\text {blur}}=$ blur score.

To increase the vessel background contrast, a familiar Contrast Limited Adaptive Histogram Equalization (CLAHE) is applied for more visualization of fine vascular structures. To increase training diversity and improve model robustness, several augmentation techniques, including horizontal flipping, rotation, brightness adjustment, and contrast variation, are employed. These preprocessing operations collectively improve vessel visibility, suppress background artifacts, and facilitate more effective vessel representation learning.

3.3 Hybrid tri-stream encoding

The proposed method incorporates a tri-stream retinal feature encoding module to extract complementary retinal vessel representations before hybrid fusion. This module consists of three parallel branches: a CNN stream for local vessel morphology extraction, a ViT stream for global contextual vessel modeling, and a texture stream for vessel pattern encoding, as in Figure 2. In the CNN branch, a ResNet backbone followed by multi-scale convolutional branches and a Feature Pyramid Network (FPN) extracts fine vessel boundaries, bifurcations, and multi-resolution vascular structures to produce a local feature map. Simultaneously, the ViT branch partitions the input image into 16 × 16 patches and applies 12-head self-attention with positional encoding to capture long-range vessel dependencies and global vascular topology, producing a global feature map. The texture branch employs a Gabor filter bank, a U-Net sub-network, and Local Binary Pattern (LBP) based texture encoding to emphasize line-like vessel patterns and local contrast variations, resulting in a texture feature map. These three complementary feature maps are then fed into the proposed Vessel-Aware Hybrid Transformer (VAHT), enabling robust multi-view vessel representation learning for accurate retinal vessel segmentation.

2.png

Figure 2. Hybrid tri-stream encoding

3.4 Multi-Scale Vision Transformer encoder

The MSVT serves as the primary global feature extraction module within the proposed framework. The preprocessed retinal image is separated into image patches and projected into a sequence of embedded tokens. Converts retinal vessel structures into token representations suitable for transformer processing, in Eq. (3).

$\left.Z_0=\left[x_p^1 E ; x_p^2 E ; \ldots \ldots \ldots ; x_p^N E\right]+E_{p o s}\right]$       (3)

where, $x_p^i=i^{t h}$ image patch, $\mathrm{E}=$ embedding matrix, $\mathrm{N}=$ number of patches, $E_{\text {pos}}=$ positional embedding, $Z_0=$ transformer input sequence.

Unlike conventional transformers operating at a single scale, the MSVT employs multiple patch resolutions to simultaneously capture coarse and fine vascular patterns as shown in Eq. (4) below.

$\operatorname{Attention}(Q, K, V)=\operatorname{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$        (4)

where, $Q=$ query matrix, $K=$ key matrix, $V=$ value matrix, $d_k=$ key dimension.

This enables effective preservation of vessel continuity, bifurcation relationships, and structural connectivity across the retinal image. The global vessel transformer output is expressed in Eq. (5).

$F_{M S V T}=M S A(Z)+M L P(Z)$       (5)

where, MSA(·) = multi-head self-attention, MLP(·) = feed-forward network, $F_{M S V T}$ = global vessel feature representation. Encodes global vessel connectivity and bifurcation relationships. Resulting transformer features provide rich contextual information that complements the vessel extraction by subsequent modules.

The VAHT-BiSRNet employs an MSVT Backbone to learn hierarchical retinal vessel representations across multiple spatial resolutions shown in Figure 3. The MSVT begins with a patch partition stem, where the image is divided into 4 × 4 patches and projected into a token sequence for transformer-based vessel encoding. Localised vessel modelling and cross-window contextual interaction are made possible by subsequent transformer stages that process the tokens utilising Window-based multi-head self-attention (W-MSA) and Shifted-window self-attention (SW-MSA). The feature resolution decreases from H/4 to H/32; the MSVT generates a vessel feature pyramid consisting of P2, P3, P4, and P5 representations. Here, P2 preserves fine vessel boundaries and capillary details, P3 and P4 encode intermediate vessel morphology and branching continuity, and P5 captures global vascular topology and long-range vessel dependencies. By jointly representing retinal vessels at multiple semantic and spatial scales, the MSVT branch strengthens vessel continuity preservation and provides robust transformer-based global context for downstream vessel-aware fusion and segmentation.

3.png

Figure 3. Multi-Scale Vision Transformer (MSVT) backbone architecture

3.5 Hybrid Feature Fusion Bidirectional Segmentation Refinement Network (HFF-Net + BiSRNet)

Transformer encoders effectively capture global contextual information; accurate vessel segmentation also requires preservation of local vessel morphology. To address this challenge, HFF-Net extracts and aggregates multi-scale vessel representations by Multi-Scale Feature Extraction, capturing both thick vessels and fine capillary structures in Eq. (6).

$F_i={Conv}_i\left(I_{a v q}\right)$       (6)

where, ${Conv}_i=$ convolution with kernel scale, $F_i$ = vessel feature map.

Eq. (7) shows that HFF-Net combines hierarchical convolutional features obtained at different receptive fields and integrates them through adaptive feature fusion mechanisms.

$F_{\text {fusion}}=\sum_{i=1}^n \alpha_i F_i$       (7)

where, $\alpha_i=$ learnable fusion weight, $F_i$ = multi-scale feature map.

This process enables simultaneous representation of thick vessels, thin capillaries, vessel crossings, and bifurcation regions.

By leveraging complementary local and contextual information, HFF-Net enhances discrimination between vascular and non-vascular regions while preserving fine vessel structures. The proposed Hybrid Feature Fusion CNN (HFF-Net) is designed to integrate the three complementary feature streams extracted from the tri-stream retinal feature encoder, namely the CNN, ViT, and texture-based vessel feature map as represented in Figure 4. In the first stage, features are spatially aligned to a common resolution of $\frac{H}{8} \times W / 8$ with 256 channels using convolutional projection, token reshaping with MLP, and 1 × 1 projection. In order to combine local and global vascular cues, the aligned features are processed through a convolutional fusion block; features are concatenated and enhanced using residual gating and depth-wise separable convolution. To further improve sensitivity to thin vessels and directional vessel patterns, an attention-weighted texture fusion module selectively injects texture features into the fused representation through a learned attention gate and vessel-mask blending mechanism. The resulting hybrid feature map is compacted using a 1 × 1 convolution and projected into a multi-scale vessel-aware output representation. The final HFF-Net output therefore provides a unified local-global-texture vessel representation that is subsequently forwarded to the VAHT/BiSRNet module for vessel-aware segmentation refinement.

To effectively combine transformer-derived contextual information and convolution-derived vessel representations, a BiSRNet is proposed. Unlike conventional one-way feature fusion approaches, BiSRNet establishes bidirectional information exchange between global and local feature streams shown in Figure 5. The bidirectional refinement mechanism progressively propagates contextual vessel information across multiple feature levels, enabling enhanced vessel continuity and structural consistency. Bidirectional feature refinement has two functions: Global-to-Local Refinement transfers global vessel topology information into local vessel representations in Eq. (8), and Local-to-Global Refinement enhances transformer features using local vessel morphology as given in Eq. (9).

$F_{G L}=F_{H F F}+\beta F_{M S V T}$        (8)

where, $F_{H F F}$ = local vessel feature, $F_{M S V T}$ = global vessel feature, β = fusion coefficient.

4.png

Figure 4. Hybrid Feature Fusion Network (HFF-Net) architecture

5.png

Figure 5. Bi-stream retinal representation network-Bidirectional Segmentation Refinement Network (BiSRNet)

$F_{L G}=F_{M S V T}+\gamma F_{H F F}$       (9)

where, $\gamma$ = refinement coefficient. Through iterative refinement, bidirectional vessel fusion preserves vessel continuity and vascular topology, as formulated in Eq. (10).

$F_{B i}=F_{G L} \oplus F_{L G}$        (10)

where, $\bigoplus$ = feature fusion operation, $F_{B i}$ = fused vessel representation.

BiSRNet reduces vessel fragmentation, improves bifurcation preservation, and strengthens connectivity among thin vascular branches. This vessel-aware fusion strategy forms one of the primary contributions of the proposed framework.

3.6 Squeeze-and-Excitation Convolutional Block Attention Module and vessel mask decoder

To further improve vessel localization, the fused feature representations are processed through a hybrid attention module that combines SE and CBAM. The SE component performs channel-wise attention learning to emphasize vessel-relevant feature channels, illustrated in Eq. (11), whereas the CBAM module sequentially applies channel and spatial attention mechanisms to highlight important vessel regions shown in Eq. (12).

$M_s=\sigma\left(f^{7 \times 7}([\operatorname{AvgPool}(F) ; \operatorname{MaxPool}(F)])\right)$       (11)

where, AvgPool = average pooling, MaxPool = max pooling, $f^{7 \times 7}$ = spatial convolution.

$F_{\text {att}}=M_c \otimes M_s \otimes F_{\text {ref}}$       (12)

where, ⊗ = element-wise multiplication.

The combined SE-CBAM strategy enables adaptive suppression of background noise, improves representation of low-contrast vessels, and enhances feature discrimination around vessel boundaries. Consequently, the attention-enhanced features provide more accurate localization of retinal vascular structures.

The final segmentation stage employs a lightweight decoder to reconstruct the vessel segmentation map from the attention-enhanced feature representations. The decoder enhances feature maps step by step, incorporating detailed vessel information via skip connections and feature refinement operations. Vessel Probability Prediction generates pixel-wise vessel probability estimates in Eq. (13).

$P_{\text {vessel}}=\sigma\left(F_{\text {dec}}\right)$        (13)

where, $F_{\text {dec}}$ = decoder output, $P_{\text {vessel}}$ = vessel probability map.

The reconstructed output is represented as a binary vessel probability map, where each pixel is classified as either vessel or background. The decoder is specifically optimized to preserve vessel thickness, maintain vascular continuity, and accurately recover complex vessel branching patterns.

3.7 Training objective

The proposed VAHT-BiSRNet is trained using a composite segmentation loss function that combines BCE and Dice loss. BCE loss optimizes pixel-wise vessel classification, in Eq. (14), while Dice loss directly maximizes overlap between predicted vessel masks as illustrated in Eq. (15) and ground-truth annotations. The total loss function simultaneously optimizes vessel boundary accuracy and segmentation overlap, given in Eq. (16). The combined objective enables balanced optimization of both vessel boundary accuracy and overall vascular structure preservation.

$L_{B C E}=\frac{1}{N} \sum_{i=1}^N\left[y_i \log \left(p_i\right)+\left(1-y_i\right) \log \left(1-p_i\right)\right]$       (14)

where, $y_i$ = ground-truth vessel label, $p_i$ = predicted vessel probability, $N$ = number of pixels.

$L_{\text {Dice}}=1-\frac{2 \sum y_i p_i+\epsilon}{\sum y_i+\sum p_i+\epsilon}$       (15)

where, $\epsilon$ = smoothing constant.

$L_{\text {Total}}=\lambda_1 L_{B C E}+\lambda_2 L_{\text {Dice}}$       (16)

where, $\lambda_1, \lambda_2$ = balancing coefficients.

Through the integration of MSVT-based global topology learning, HFF-Net multi-scale vessel representation, BiSRNet bidirectional fusion, and SE-CBAM attention enhancement, the proposed methodology achieves accurate and computationally efficient retinal vessel segmentation across the benchmark datasets.