Brain-Inspired Reliability-Guided Multi-Level Interaction Network for Infrared–Visible Image Fusion

Infrared and visible images describe the same scene through different physical sensing mechanisms. Visible-light imaging records reflected radiation and generally provides rich textures, edges, and structural details under favorable illumination, whereas infrared imaging responds to thermal radiation and can preserve target contrast under darkness, uneven illumination, or partial visual degradation. Infrared–visible image fusion (IVIF) therefore seeks to integrate complementary information from the two modalities into a single representation that retains both thermally salient targets and visually informative structures [1-4]. Such fusion has become relevant to surveillance, autonomous perception, object detection, and other vision tasks in which a single sensing modality may not provide sufficiently complete scene information.

Early IVIF methods relied mainly on multi-scale transforms, sparse representation, saliency analysis, subspace modeling, and manually designed fusion rules [4, 5]. Although these approaches offer relatively transparent processing mechanisms, their performance depends strongly on the selected decomposition strategy, activity measure, and fusion rule. Deep learning subsequently shifted IVIF toward data-driven feature representation. DenseFuse introduced a densely connected encoder–decoder architecture for latent feature extraction and reconstruction [6], FusionGAN formulated fusion through adversarial image generation [7], and U2Fusion developed a unified unsupervised framework for multiple image-fusion tasks [8]. RFN-Nest and DIDFuse further explored residual feature fusion and deep image decomposition, respectively [9, 10]. These developments established the basic learning-based paradigm of extracting modality-specific features and reconstructing a fused image from learned representations.

More recent studies have increasingly focused on adaptive information preservation rather than uniform feature aggregation. PIAFusion introduced illumination-aware modeling to adapt fusion behavior to changing lighting conditions [11], MetaFusion incorporated object-detection-derived meta-features into the fusion process [12], and SGFusion used saliency guidance to control information preservation [13]. MUFusion introduced memory-based unsupervised representation [14], while adaptive pixel-weighting strategies directly estimated spatially varying fusion weights for efficient infrared–visible integration [15]. These methods demonstrate that the relative importance of infrared and visible information is strongly dependent on local scene content and cannot be adequately described by a fixed global fusion rule.

A related challenge concerns how complementary information should be exchanged across representation levels. Shallow features mainly retain gradients, edges, and local textures, whereas deeper features progressively encode larger structures and salient content. Interactive Feature Embedding explicitly modeled information exchange between infrared and visible representations [16]. CoCoNet combined coupled contrastive learning with multi-level feature ensembles [17], MFIFusion introduced multi-level feature injection [18], and MDAN employed a multilevel dual-branch attention architecture [19]. BCMFIFuse further modeled bilateral cross-modal interaction in spatial and channel dimensions [20]. Transformer-based approaches extended this direction through progressive token exchange and interactive attention [21-23]. More recently, semantic interaction and selective state-space modeling have been incorporated into fusion networks, as illustrated by SFINet, S4Fusion, and CrossMamba [24-26]. Collectively, these studies show that hierarchical cross-modal interaction is important for preserving complementary information at different spatial and semantic scales.

However, effective interaction alone does not resolve a more fundamental issue: not every strong feature response is equally reliable. Infrared responses may dominate around thermally salient targets, whereas visible features may be more informative around road markings, building boundaries, vegetation, and fine textures. At the same time, thermal saturation, local sensor noise, low illumination, or overexposure can also generate strong activations that should not necessarily be transferred to the opposite modality. Conventional attention mechanisms generally estimate feature importance from continuous activations, but feature magnitude and modality reliability are not equivalent. Unrestricted information exchange may therefore propagate locally unreliable responses even when cross-modal interaction itself is well designed.

Spiking neural networks (SNNs) provide an alternative representation mechanism based on temporal membrane-state accumulation and threshold-triggered firing [27-29]. In a leaky integrate-and-fire neuron, incoming stimulation is accumulated in the membrane potential, while spike events are produced only when the neuronal state exceeds a firing threshold. The membrane potential therefore retains sub-threshold responses that may remain informative even when no spike is generated. This property is particularly relevant to image fusion, where weak edges and low-contrast textures should not be discarded simply because their responses are less pronounced than those of thermally salient targets. Recent work has also demonstrated that SNN-based architectures can be applied directly to IVIF, confirming the relevance of membrane dynamics and spike activity to multimodal fusion [30].

Nevertheless, using a fully spiking network as the principal reconstruction backbone is not necessarily desirable for detail-sensitive fusion. Repeated spike quantization may suppress weak but perceptually important image structures, whereas continuous-valued features remain advantageous for fine-grained intensity and texture reconstruction. This motivates a different division of responsibility: continuous neural representations can be retained for image-content encoding and reconstruction, while spiking dynamics can be used as an auxiliary mechanism for estimating local modality reliability.

Based on this consideration, a Brain-Inspired Reliability-Guided Multi-Level Interaction Network (BMIF-Net) is proposed for IVIF. The framework separates continuous image representation from spiking-based reliability estimation. A dual-branch multi-scale encoder first extracts modality-specific features at three representation levels. A Spiking Modality Reliability Estimator (SMRE) then characterizes local modality responses through leaky integrate-and-fire dynamics and derives reliability cues from both membrane-state accumulation and firing activity. These cues are subsequently introduced into a Reliability-Modulated Multi-Level Cross-Modal Interaction (RMCI) module to regulate bidirectional feature exchange. A Reliability-Guided Adaptive Fusion (RGAF) module further combines interacted features and reliability cues to determine spatially varying modality contributions before multi-level reconstruction.

The main contributions of this study are summarized as follows:

(1) A hybrid continuous–spiking fusion architecture is developed in which continuous neural features remain responsible for detailed image representation and reconstruction, while spiking dynamics are used specifically for modality reliability estimation. This design avoids forcing the complete fusion pathway into a discrete spiking representation.

(2) A SMRE is introduced to characterize local modality reliability using both membrane-state accumulation and firing activity. The joint representation preserves continuous sub-threshold responses while retaining threshold-driven neuronal saliency.

(3) A RMCI module is developed to regulate complementary information transfer according to the reliability of the contributing modality. This reduces the propagation of locally unreliable responses during feature interaction.

(4) A RGAF module combines interacted continuous features with spiking-derived reliability cues to generate spatially varying modality weights, allowing the preferred information source to change with local scene content.

(5) The proposed framework is evaluated using quantitative comparison, component ablation, reliability-representation analysis, temporal simulation analysis, and computational-complexity assessment. The experiments examine overall fusion performance together with the contributions of cross-modal interaction, the complete reliability-guided framework, and the incorporation of spike activity into the reliability representation.

3.1 Overall architecture

The proposed BMIF-Net integrates continuous image representation with spiking-based modality reliability estimation. The network consists of five main components: a dual-branch multi-scale encoder, a SMRE, a RMCI module, a RGAF module, and a lightweight multi-level reconstruction decoder. The overall architecture is illustrated in Figure 1.

1.png

Let $I_{i r}$ and $I_{v i s}$ denote a registered infrared image and visible image, respectively. For each representation level $l$, the complete fusion process can be expressed as

$I_f=\mathcal{D}\left(\left\{\mathcal{G}^l\left(\mathcal{M}^l\left(F_{i r}^l, F_{v i s}^l ; R_{i r}^l, R_{v i s}^l\right) ; R_{i r}^l, R_{v i s}^l\right)\right\}_{l=1}^3\right)$       (1)

where, $\mathcal{D}, \mathcal{G}^l$, and $\mathcal{M}^l$ denote the decoder, RGAF, and RMCI at level $l$, respectively, while $R_{i r}^l$ and $R_{v i s}^l$ are the modality reliability maps estimated by SMRE.

The dual encoders extract three levels of modality-specific features:

$\left\{F_m^l\right\}_{l=1}^3=\varepsilon_m\left(I_m\right), m \in\{i r, v i s\}$       (2)

The overall information flow is therefore summarized as

$\begin{aligned}\left(I_{i r}, I_{v i s}\right) \rightarrow\left\{F_{i r}^l,\right. & \left.F_{v i s}^l\right\} \rightarrow\left\{R_{i r}^l, R_{v i s}^l\right\} \rightarrow\left\{\tilde{F}_{i r}^l, \tilde{F}_{v i s}^l\right.\rightarrow\left\{F_f^l\right\} \rightarrow I_f\end{aligned}$       (3)

Reliability estimation is performed before extensive cross-modal mixing. This ordering is deliberate: locally unreliable responses are first identified and can subsequently be attenuated before they propagate through cross-modal interaction and final fusion.

3.2 Dual-branch multi-scale feature encoder

Infrared and visible images exhibit different intensity distributions, structural characteristics, and modality-specific response patterns. A fully shared encoder may restrict the ability of the network to specialize for these heterogeneous sensing mechanisms. BMIF-Net therefore employs two encoder branches with identical architectures but independent learnable parameters.

For modality $m \in\{i r, v i s\}$, the encoder produces three hierarchical feature representations:

$F_m^1 \in \mathbb{R}^{H \times W \times 32}, F_m^2 \in \mathbb{R}^{\frac{H}{2} \times \frac{W}{2} \times 64}, F_m^3 \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times 12}$       (4)

The first level retains fine spatial information, including gradients, local boundaries, and textures. The second level represents larger local structures, whereas the third level provides broader contextual and salient information.

Each level contains a residual feature-extraction block consisting of two $3 \times 3$ convolutions. The second and third levels are preceded by stride-2 convolutions that simultaneously reduce the spatial resolution and increase the channel dimension. Only two downsampling operations are employed because excessive spatial reduction may suppress weak boundaries and low-contrast structures that remain important for fusion.

Although the infrared and visible branches have the same topology, their parameters are independent. This allows the infrared encoder to specialize in thermal contrast and target-related responses, while the visible encoder can retain greater sensitivity to edges, textures, and structural details.

Before the extracted features enter SMRE, each feature level is projected through a learnable $1 \times 1$ convolution:

$Z_m^l=P_m^l\left(F_m^l\right), l \in\{1,2,3\}$       (5)

where, $P_m^l$ denotes a learnable $1 \times 1$ convolutional projection and $Z_m^l$ is the representation supplied to the spiking reliability estimator. The projection preserves the spatial resolution and channel dimension of the corresponding encoder feature.

This design separates two responsibilities. Continuous features retain detailed image content for interaction and reconstruction, whereas the spiking branch operates on their projected representations to estimate local modality reliability.

3.3 Spiking Modality Reliability Estimator

SMRE estimates how reliably each modality responds to local scene content through leaky integrate-and-fire neuronal dynamics. Rather than replacing the continuous feature representation with discrete spikes, the module uses neuronal state evolution as an auxiliary source of reliability information. Its structure is illustrated in Figure 2.

2.png

Figure 2. Detailed structure of the Spiking Modality Reliability Estimator (SMRE)

Because the source images are static, the projected representation is repeatedly supplied to the LIF neurons during a short simulation window:

$Z_{m, t}^l=Z_m^l, t=1,2, \ldots, T$       (6)

The membrane potential is updated according to

$U_{m, t}^l=\beta U_{m, t-1}^l+Z_m^l-V_{\mathrm{th}} S_{m, t-1}^l$      (7)

where, $U_{m, t}^l$ denotes the membrane potential at simulation step $t$, $\beta$ is the membrane decay factor, $V_{\mathrm{th}}$ is the firing threshold, and $S_{m, t-1}^l$ represents the firing state at the preceding step. The final term implements reset by subtraction following neuronal firing.

The firing state is determined by

$S_{m, t}^l=H\left(U_{m, t}^l-V_{\mathrm{th}}\right)$      (8)

where, $H(\cdot)$ is the threshold function. Its non-differentiable derivative is replaced by a smooth surrogate gradient during backpropagation, enabling joint optimization of the spiking and continuous components.

After $T$ simulation steps, the membrane state and firing activity are temporally averaged:

$\bar{U}_m^l=\frac{1}{T} \sum_{t=1}^T U_{m, t}^l, \bar{S}_m^l=\frac{1}{T} \sum_{t=1}^T S_{m, t}^l$       (9)

The averaged membrane state preserves continuous accumulated evidence, including sub-threshold responses, while the averaged spike activity represents the frequency with which local responses exceed the neuronal firing threshold.

The two representations are jointly used to construct the modality reliability map:

$R_m^l=\sigma\left[C_r^l\left(\left[\mathcal{N}\left(\bar{U}_m^l\right), \bar{S}_m^l\right]\right)\right]$      (10)

where, $\mathcal{N}(\cdot)$ denotes channel-wise normalization of the mean membrane state, $C_r^l$ denotes a learnable $1 \times 1$ convolution, $[\cdot, \cdot]$ denotes channel concatenation, and $\sigma(\cdot)$ denotes the sigmoid function.

Accordingly,

$0<R_m^l(x, y)<1$      for every spatial position $(x, y)$      (11)

A larger value represents a stronger learned reliability response derived jointly from accumulated membrane evidence and firing behavior. The resulting map is used as a relative control signal for subsequent feature interaction and fusion rather than as a physically calibrated sensor-confidence probability. Importantly, the membrane-state component prevents the reliability estimator from depending exclusively on binary firing events. Weak but persistent structures can therefore contribute to reliability estimation even when they do not repeatedly cross the firing threshold.

3.4 Reliability-Modulated Multi-Level Cross-Modal Interaction

RMCI exchanges complementary infrared and visible information while preventing features from being transferred indiscriminately. Its detailed structure is illustrated in Figure 3.

3.png

Figure 3. Detailed structure of the Reliability-Modulated Multi-Level Cross-Modal Interaction (RMCI) module

At each feature level, the infrared feature, visible feature, and their absolute difference are first concatenated:

$D^l=\left[F_{i r}^l, F_{v i s}^l,\left|F_{i r}^l-F_{v i s}^l\right|\right]$       (12)

The difference term explicitly represents disagreement between the two modalities and allows the interaction module to detect regions where their information differs substantially.

Channel interaction is estimated from globally pooled features:

$A_c^l=\sigma\left[W_{c 2}^l\left(\delta\left[W_{c 1}^l\left(\operatorname{GAP}\left(D^l\right)\right)\right]\right)\right]$        (13)

where, GAP denotes global average pooling, $W_{c 1}^l$ and $W_{c 2}^l$ denote learnable $1 \times 1$ convolutional mappings, and ReLU is the rectified linear activation function.

A spatial interaction map is obtained from channel-wise average and maximum responses:

$A_s^l=\sigma\left[C_s^l\left(\left[\operatorname{Mean}_c\left(D^l\right), \operatorname{Max}_s\left(D^l\right)\right]\right)\right]$       (14)

where, channel-wise average and maximum pooling are applied to $D^l$, and the resulting two-channel spatial descriptor is processed by a $7 \times 7$ convolution followed by a sigmoid activation.

The candidate information transferred from the visible branch to the infrared branch is

$Q_{v i s \rightarrow i r}^l=\Phi_{v i s \rightarrow i r}^l\left(D^l\right) \odot A_c^l \odot A_s^l$       (15)

while the reverse-direction complementary information is

$Q_{i r \rightarrow v i s}^l=\Phi_{i r \rightarrow v i s}^l\left(D^l\right) \odot A_c^l \odot A_s^l$      (16)

where, $\Phi_{v i s \rightarrow i r}^l$ and $\Phi_{i r \rightarrow v i s}^l$ are independent learnable $3 \times 3$ convolutional mappings and $\odot$ represents element-wise multiplication.

The reliability of the contributing modality subsequently regulates the amount of information transferred:

$\tilde{F}_{i r}^l=F_{i r}^l+R_{v i s}^l \odot Q_{v i s \rightarrow i r}^l$       (17)

$\tilde{F}_{v i s}^l=F_{v i s}^l+R_{i r}^l \odot Q_{i r \rightarrow v i s}^l$       (18)

The residual formulation retains the modality-specific representation and adds only a reliability-weighted complementary component. Consequently, a locally unreliable visible response contributes less to the infrared representation, and vice versa.

This distinction between feature interaction and modality reliability is central to RMCI. Cross-modal relevance determines what information is potentially useful, whereas the independently estimated reliability map determines how strongly that information should influence the opposite modality.

3.5 Reliability-Guided Adaptive Fusion

After cross-modal interaction, RGAF determines the relative contributions of the two interacted modalities to the fused representation. Its structure is illustrated in Figure 4.

4.png

Figure 4. Detailed structure of the Reliability-Guided Adaptive Fusion (RGAF) module

At representation level $l$, the interacted features, reliability maps, and absolute reliability difference are concatenated:

$X^l=\left[\tilde{F}_{i r}^l, \tilde{F}_{v i s}^l, R_{i r}^l, R_{v i s}^l,\left|R_{i r}^l-R_{v i s}^l\right|\right]$       (19)

The difference between $R_{i r}^l$ and $R_{v i s}^l$ explicitly represents local disagreement in modality confidence.

A lightweight convolutional predictor generates the infrared contribution map:

$A_{i r}^l=\sigma\left[C_2^l\left(\delta\left[C_1^l\left(X^l\right)\right]\right)\right], A_{v i s}^l=1-A_{i r}^l$       (20)

where, $C_1^l$ and $C_2^l$ are learnable convolutional mappings.

The final fused feature at level $l$ is

$F_f^l=A_{i r}^l \odot \widetilde{F}_{i r}^l+A_{v i s}^l \odot \widetilde{F}_{v i s}^l$       (21)

Because the weighting map varies spatially, the preferred modality can change from one image region to another. Thermally salient targets may receive a larger infrared contribution, whereas texture-rich regions can retain a stronger visible contribution.

The fusion weights are not determined by reliability maps alone. The interacted continuous features are also incorporated into Eq. (19), ensuring that the final fusion decision depends jointly on local modality confidence and actual feature content.

3.6 Multi-level reconstruction

The decoder reconstructs the fused image from the three fused feature levels in a top-down manner. The deepest fused representation is first transformed as

$G^3=\mathcal{D}_3\left(F_f^3\right)$       (22)

where, $\mathcal{D}_3$ consists of a $3 \times 3$ convolution, a ReLU activation, and a residual feature block.

The second-level reconstruction combines the corresponding fused representation with the upsampled deeper feature:

$G^2=\mathcal{D}_2\left(\left[F_f^2, \operatorname{Up}\left(G^3\right)\right]\right)$       (23)

The full-resolution representation is reconstructed as

$G^1=\mathcal{D}_1\left(\left[F_f^1, \operatorname{Up}\left(G^2\right)\right]\right)$        (24)

where, $\operatorname{Up}(\cdot)$ denotes bilinear interpolation to the spatial resolution of the corresponding shallower feature map.

Finally, a $1 \times 1$ convolution followed by a sigmoid activation produces the fused image:

$I_f=\sigma\left[C_{\text {out }}\left(G^1\right)\right]$      (25)

This hierarchical reconstruction process retains high-level contextual information while progressively restoring local boundaries, textures, and fine spatial structures from shallower fused representations.

3.7 Training objective

A unique ground-truth fused image is generally unavailable for IVIF. BMIF-Net is therefore trained using complementary unsupervised constraints designed to preserve source intensity, gradients, and structural information.

The intensity loss encourages the fused image to preserve locally prominent source responses:

$\mathcal{L}_{\text {int}}=\left\|I_f-\max \left(I_{i r}, I_{\text {vis}}\right)\right\|_1$       (26)

where, the maximum operation is performed element-wise.

The gradient loss encourages preservation of strong boundaries and local structural details:

$\mathcal{L}_{\mathrm{grad}}=\left\|\nabla I_f-\max \left(\nabla I_{i r}, \nabla I_{v i s}\right)\right\|_1$       (27)

where, ∇ denotes the gradient-magnitude operator calculated from horizontal and vertical finite differences.

Structural consistency with both source modalities is further encouraged using

$\mathcal{L}_{s s i m}=1-\frac{1}{2}\left[\operatorname{SSIM}\left(I_f, I_{i r}\right)+\operatorname{SSIM}\left(I_f, I_{v i s}\right)\right]$       (28)

The complete optimization objective is

$\mathcal{L}=\lambda_{\text {int}} \mathcal{L}_{\text {int}}+\lambda_{\text {grad}} \mathcal{L}_{\text {grad}}+\lambda_{\text {ssim}} \mathcal{L}_{\text {ssim}}$      (29)

where, $\lambda_{\text {int}}, \lambda_{\text {grad}}$, and $\lambda_{\text {ssim}}$ control the relative contributions of intensity preservation, gradient preservation, and structural consistency. Their values are specified in Section 4.2 and are kept unchanged throughout the reported experiments.

5.1 Quantitative comparison

The quantitative performance of BMIF-Net is first evaluated on the MSRS test set using seven complementary fusion metrics: EN, SD, SF, MI, VIF, QAB/F, and SCD. All methods are evaluated using the same test pairs and the same metric implementation. The results are reported in Table 1.

Table 1. Quantitative comparison on the MSRS

BMIF-Net achieves an EN of 6.4131, an SD of 41.7063, an SF of 10.9309, an MI of 2.9500, a VIF of 2.1077, a QAB/F of 0.6369, and an SCD of 1.6084. Its most pronounced advantage is observed in VIF, where the proposed method substantially exceeds all four classical baselines. Compared with the strongest classical VIF result of 1.0406 obtained by Max fusion, BMIF-Net increases VIF to 2.1077. This indicates that the proposed framework preserves substantially more perceptually relevant information from the two source modalities.

The proposed method also achieves strong SF, QAB/F, and SCD values. Its SF of 10.9309 is very close to the highest value of 10.9924 obtained by Max fusion, indicating that BMIF-Net preserves a high level of spatial variation and fine detail. The QAB/F value of 0.6369 and SCD value of 1.6084 further indicate effective preservation of edge information and complementary source content.

BMIF-Net does not achieve the highest EN or MI. Max fusion obtains a higher EN, while PCA produces the highest MI. These differences are expected because individual metrics emphasize different statistical properties. For instance, PCA directly maximizes variance-related information and can therefore produce a very high MI value without necessarily yielding stronger perceptual fidelity. Similarly, Max fusion favors locally stronger intensities and may increase entropy or edge-related measures.

The performance of BMIF-Net is therefore better interpreted from a multi-metric perspective. Its dominant VIF and competitive SF, QAB/F, and SCD values indicate that the proposed reliability-guided mechanism places greater emphasis on perceptual fidelity, structural preservation, and complementary information transfer rather than on maximizing a single statistical criterion.

5.2 Core component ablation study

Ablation experiments are conducted on the MSRS test set to examine the contribution of the proposed interaction and reliability-guided fusion mechanisms. All configurations use the same dual-branch encoder, reconstruction decoder, training set, optimizer, objective function, training schedule, and evaluation protocol. Only the fusion mechanism is changed.

The Baseline configuration removes SMRE, RMCI, and RGAF. Infrared and visible features are directly averaged at each representation level:

$F_f^l=\frac{1}{2}\left(F_{i r}^l+F_{v i s}^l\right)$       (32)

This configuration provides a reference for evaluating whether the proposed interaction and reliability-guided modules contribute beyond the underlying encoder-decoder architecture.

The RMCI-only configuration removes SMRE and RGAF. To isolate the contribution of cross-modal interaction, the two reliability maps are fixed to one:

$R_{i r}^l=R_{v i s}^l=1$       (33)

RMCI therefore performs bidirectional information exchange without reliability modulation. The interacted features are subsequently averaged:

$F_f^l=\frac{1}{2}\left(\tilde{F}_{i r}^l+\tilde{F}_{v i s}^l\right)$       (34)

The Full BMIF-Net retains SMRE, RMCI, and RGAF. SMRE estimates modality reliability, RMCI uses the reliability maps to regulate bidirectional feature exchange, and RGAF generates spatially varying fusion weights according to Eq. (21).

The quantitative results are shown in Table 2.

Table 2. Core component ablation on MSRS

Introducing RMCI improves SF from 9.6709 to 10.0537, MI from 2.9028 to 2.9565, VIF from 1.9584 to 1.9866, QAB/F from 0.5638 to 0.5957, and SCD from 1.5754 to 1.5776. These changes demonstrate that explicit cross-modal interaction facilitates the exchange of complementary infrared and visible information.

The full model produces a larger improvement in the metrics most closely associated with structural and perceptual preservation. Relative to the Baseline, SF increases from 9.6709 to 10.9309, VIF from 1.9584 to 2.1077, QAB/F from 0.5638 to 0.6369, and SCD from 1.5754 to 1.6084.

The comparison between the RMCI-only configuration and the complete model shows that introducing the full reliability-guided fusion framework provides additional gains beyond unrestricted cross-modal interaction. In the complete model, spiking-derived reliability information is used to modulate bidirectional feature transfer and to support spatially adaptive modality weighting. The improvements in VIF, QAB/F, and SCD therefore support the combined contribution of reliability estimation and reliability-guided fusion, although the individual effects of SMRE and RGAF are not isolated separately in the present ablation design.

5.3 Analysis of spiking reliability representation

The reliability representation used in SMRE is further examined to determine whether spike activity provides useful information beyond continuous membrane responses.

In the membrane-only configuration, the reliability map is estimated exclusively from the normalized average membrane potential:

$R_{m, U}^l=\sigma\left[C_{r, U}^l\left(\mathcal{N}\left(\bar{U}_m^l\right)\right)\right]$       (35)

The complete SMRE jointly uses the average membrane potential and average firing activity:

$R_m^l=\sigma\left[C_r^l\left(\left[\mathcal{N}\left(\bar{U}_m^l\right), \bar{S}_m^l\right]\right)\right]$      (36)

The corresponding results are presented in Table 3.

Table 3. Effect of incorporating spike activity into SMRE reliability representation on MSRS

The membrane-only representation achieves higher EN and MI, indicating that continuous membrane states retain strong statistical information from the encoded features. However, adding spike activity improves VIF from 2.0991 to 2.1077, QAB/F from 0.6329 to 0.6369, and SCD from 1.5959 to 1.6084.

Membrane potential and firing activity therefore capture complementary aspects of modality responsiveness. The membrane state accumulates continuous input evidence and therefore retains sub-threshold responses, whereas spike activity emphasizes responses that repeatedly exceed the neuronal firing threshold. Combining the two representations slightly shifts the fusion behavior away from maximizing statistical information content and toward stronger perceptual fidelity, edge preservation, and complementary structural transfer.

The result also clarifies the role of spiking computation in BMIF-Net. Spike activity does not uniformly improve every metric, nor is it used to replace continuous image representation. Instead, the present comparison indicates that incorporating spike activity in addition to membrane-state information provides an additional reliability cue and improves several metrics associated with perceptual and structural fusion quality. Because a spike-only configuration is not evaluated, the experiment is intended to assess the incremental contribution of spike activity rather than to provide a complete comparison among alternative reliability representations.

5.4 Effect of the spiking simulation length

The simulation length $T$ determines the number of neuronal-state updates performed by SMRE. A longer simulation window allows additional membrane accumulation and firing events but also increases computational cost. To examine the effect of a short temporal simulation window on fusion performance, two representative settings are compared:

$T \in\{2,4\}$       (37)

All remaining model and training parameters are kept unchanged.

The quantitative results are shown in Table 4.

Table 4. Effect of the spiking simulation length on MSRS

The differences between the two settings are relatively small. With $T$ = 2, the model achieves slightly higher EN, VIF, and SCD, whereas $T$ = 4 produces a marginally higher QAB/F.

The limited variation across the two settings shows that useful reliability information can be obtained within a short simulation window. Even two state updates are sufficient to generate useful membrane and firing responses. Increasing the simulation length to four steps does not produce a uniform improvement across all metrics, but it provides a moderately longer response-accumulation window and slightly stronger edge preservation.

The default configuration therefore uses

$T=4$       (38)

as a moderate temporal setting for the reliability-estimation process. This choice retains a short simulation window while allowing both membrane accumulation and spike activity to contribute to reliability estimation.

The results should not be interpreted as evidence that larger values of $T$ necessarily improve fusion quality. Instead, they indicate that satisfactory reliability estimation can be achieved with a small number of neuronal-state updates, which is favorable for controlling the additional computational cost introduced by the spiking branch.

5.5 Computational complexity

The computational cost of BMIF-Net is evaluated in terms of trainable parameters and floating-point operations. The results are reported in Table 5.

Table 5. Computational complexity of BMIF-Net

For an input resolution of 128 × 128, BMIF-Net contains approximately 3.256 million trainable parameters and requires approximately 7.843 GFLOPs.

The parameter count remains moderate despite the use of dual modality-specific encoders, three-level reliability estimation, cross-modal interaction, and adaptive fusion. This is partly because SMRE operates on projected intermediate representations and generates single-channel spatial reliability maps rather than introducing a second large reconstruction backbone.

The additional computation associated with the brain-inspired component mainly arises from repeated LIF-state updates over $T$ simulation steps. RMCI further introduces channel and spatial interaction operations at three representation levels. Nevertheless, these modules operate on hierarchical intermediate features and therefore do not result in excessive model growth.

The reported FLOPs should be interpreted together with the temporal nature of the spiking branch. Conventional FLOP-counting tools primarily account for convolutional and matrix operations and may not fully represent the cost of element-wise membrane-state updates. For this reason, the present analysis uses parameter count and standard FLOPs primarily as architecture-level complexity indicators rather than as hardware-specific runtime measures.

Overall, the complexity results show that BMIF-Net maintains a relatively compact model size while supporting explicit reliability estimation and hierarchical cross-modal interaction. This provides a practical balance between representational capability and computational cost for IVIF.

5.6 Summary of experimental findings

The experiments provide three main observations.

First, BMIF-Net demonstrates its clearest quantitative advantage in perceptual information fidelity. On MSRS, the proposed method achieves a VIF of 2.1077, substantially exceeding the classical baselines, while maintaining competitive SF, QAB/F, and SCD values. This indicates that the network preserves a strong balance between perceptually meaningful information, spatial detail, and complementary source content.

Second, the ablation results show that multi-level cross-modal interaction improves the baseline model, while the complete reliability-guided framework provides further gains in SF, VIF, QAB/F, and SCD. The results therefore support the use of reliability information not only during final modality weighting but also during cross-modal information transfer.

Third, the reliability-representation analysis indicates that membrane states and spike activity play complementary roles. Continuous membrane information provides strong statistical representation, whereas adding firing activity improves several structural and perceptual metrics. The temporal analysis further shows that useful reliability estimates can be obtained using a relatively short neuronal simulation.

Taken together, these results support the central design principle of BMIF-Net: spiking dynamics are most useful in this framework as a reliability-estimation mechanism that controls continuous multimodal information flow, rather than as a replacement for continuous image representation and reconstruction.

This study presents BMIF-Net, a BMIF-Net for IVIF. The framework separates continuous image-content representation from spiking-based modality reliability estimation. Continuous convolutional features are retained for detailed image encoding and reconstruction, while membrane-state dynamics and spike activity are used to estimate local modality reliability.

The proposed SMRE module derives reliability information from temporally averaged membrane potentials and firing activity. These reliability cues are subsequently introduced into RMCI to regulate bidirectional cross-modal feature transfer and into RGAF to support spatially adaptive modality weighting. In this way, spiking dynamics function as a control mechanism for heterogeneous information flow rather than as a replacement for continuous image representation.

Experiments on the MSRS dataset show that BMIF-Net achieves an EN of 6.4131, an SD of 41.7063, an SF of 10.9309, an MI of 2.9500, a VIF of 2.1077, a QAB/F of 0.6369, and an SCD of 1.6084. The clearest quantitative advantage is observed in VIF, while the proposed method also maintains competitive performance in spatial-detail preservation, edge-information retention, and complementary-information transfer.

The ablation experiments further show that explicit cross-modal interaction improves the baseline model and that the complete reliability-guided framework provides additional gains beyond unrestricted interaction. The reliability-representation analysis indicates that membrane-state information and spike activity provide complementary cues: membrane potentials preserve accumulated sub-threshold evidence, whereas spike activity contributes additional threshold-driven information for reliability estimation.

The temporal analysis shows that useful reliability estimates can be obtained with a short neuronal simulation, with relatively small differences between T = 2 and T = 4. The default setting of T = 4 therefore serves as a moderate operating point rather than an empirically dominant optimum. In terms of model complexity, BMIF-Net contains approximately 3.256 million trainable parameters and requires approximately 7.843 GFLOPs for an input resolution of 128 × 128.

Several limitations remain. The current framework assumes spatially registered infrared and visible inputs, estimates modality reliability from learned feature responses rather than explicit physical uncertainty, and evaluates temporal behavior using only two representative simulation lengths. In addition, the present quantitative comparison focuses on fully reproducible classical baselines and controlled ablation experiments. Broader evaluation against recent deep-learning-based fusion methods under a unified implementation would further strengthen the empirical assessment.

Future work will investigate joint registration and fusion, uncertainty-aware reliability modeling, more efficient event-driven or adaptive temporal computation, and broader evaluation across heterogeneous multimodal fusion scenarios.

Overall, BMIF-Net demonstrates that spiking dynamics can be effectively incorporated into a continuous image-fusion architecture as a selective reliability-estimation mechanism. By combining spiking-derived reliability cues with reliability-modulated interaction and adaptive fusion, the framework provides a practical approach for preserving perceptually meaningful and structurally complementary information in infrared–visible image fusion.