Muthna J. Fadhil*
| F. Richard Yu | Haider W. Oleiwi
© 2026 The authors. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Massive multiple-input multiple-output (MIMO) is a key technology for future wireless systems, which enables improved spectrum efficiency and system capacity. However, its performance is seriously restricted by hardware imperfections such as phase noise, IQ imbalance, power amplifier (PA) nonlinearity, and erroneous channel state information (CSI) in practical applications. Such distortions cannot be fully eliminated by standard linear detectors such as maximum ratio combining (MRC), zero-forcing (ZF), and minimum mean square error (MMSE), especially in high signal-to-noise ratio (SNR) regimes. This paper proposes a hybrid AI-assisted receiver, combining an adaptive fusion process, an extreme neural network-based error correction module, and linear MMSE detection to address these challenges. The proposed technique exploits both the nonlinear compensation ability of artificial intelligence and the robustness of linear detection to improve the detection accuracy in practical scenarios. Simulation results show that the proposed receiver greatly decreases the bit error rate (BER) compared with conventional methods. The full hybrid receiver reduces the BER from about 10-3 to 10-6 at SNR = 30dB, and at high SNR the reductions are up to two orders of magnitude. Moreover, the proposed technique retains a favorable performance-complexity trade-off while showing high robustness to hardware restrictions and CSI errors. These results confirm the usefulness of the proposed hybrid architecture for reliable communication in future 6G massive MIMO systems.
AI-assisted detection, bit error rate optimization, hardware impairments, channel estimation, hybrid receiver, massive multiple-input multiple-output, wireless communications, 6G
Wireless communication systems future generation, such as 5G and beyond 6G networks, massive multiple-input multiple-output (MIMO) has become a crucial enabling technology [1-4]. Massive MIMO systems may greatly increase system capacity, energy efficiency, and spectral efficiency by using a large number of antennas at the base station [5-7]. However, in real-world implementations, a number of hardware impairments, including phase noise, IQ imbalance, power amplifier (PA) nonlinearity, and inaccurate channel state information (CSI), frequently restrict the performance of massive MIMO systems [8-11]. Conventional linear detectors are unable to completely reduce the nonlinear distortions and residual interference caused by these deficiencies [12, 13]. Because of this, popular methods like minimum mean square error (MMSE), zero-forcing (ZF), and maximum ratio combining (MRC) experience performance loss, especially in high signal-to-noise ratio (SNR) regimes where error floors are frequently seen [14, 15].
Several approaches have been proposed in the past few years to mitigate the impact of hardware impairments in massive MIMO systems. Because of their low computing complexity and scalability, conventional linear detectors such as MRC, ZF and MMSE are often utilized. For instance, a complete investigation on the efficiency of linear receivers in large-scale MIMO systems was presented in the study [16]. The efficiency and the practical constraints of the linear receivers were highlighted. Similarly, at moderate interference levels, the study in study [17] showed that MMSE-based detection was more robust than ZF and MRC. Conversely, nonlinear detection methods have been researched to achieve almost optimum performance. Maximum likelihood (ML) detection, as examined in the study [18], has the best detection accuracy, but its exponential processing complexity makes it impractical for large MIMO systems.
Sphere decoding techniques were developed in the study [19] to lower complexity; nevertheless, in large-scale applications, their processing load remains substantial. Artificial intelligence (AI)-based techniques have been introduced more recently to address nonlinear distortions and uncertainty in models. Deep learning-based receivers, such as those proposed in study [15], have demonstrated the ability to learn complex channel characteristics and improve detection accuracy. Furthermore, data-driven detection systems studied in study [20] have shown potential performance increases when taking into account hardware restrictions. However, these approaches frequently need enormous training sets and substantial computing power, which limits their practical usage.
Although nonlinear detection approaches can achieve superior performance [12, 13], they are less appropriate for large-scale massive MIMO systems as they frequently need considerably larger computer complexity. However, the goal of our study is to find a promising trade-off between productivity and computational complexity [12] by employing AI-based refinement to improve conventional linear detection. Advanced signal processing approaches and reliable detection algorithms have been developed [21-23] for better system performance under hardware restrictions. Recently, interest in the capacity of AI and deep learning-based receivers to understand complicated nonlinear connections and adjust for distortions has been raised [24-27]. Fully AI-based receivers are intriguing, but their interpretability, vast training datasets, and high computational complexity limit their practical implementation [28]. The crowd needs a balanced approach that combines the resilience of AI-based techniques and the efficiency and convenience of use of conventional linear detectors. In this context, hybrid receiver designs, where AI is employed to improve traditional signal processing instead of replacing it, have lately emerged as a potential option [29, 30]. In recent years, a variety of novel architectures have emerged in the field of AI-based wireless receivers, some of which have garnered significant interest. In addition to the traditional deep learning architectures, there have been several promising architectures have emerged in the realm of AI-based wireless receivers in recent years. In particular, deep unfolding methods have shown the capability of combining model-based optimization with neural networks to achieve better detection accuracy, while maintaining the interpretability of the algorithm [31]. In addition, graph neural network (GNN) based methods have been studied to take advantage of the structure of user-to-antennas and user-to-user relationships to improve beamforming and signal detection in multi-user MIMO (MU-MIMO) systems [32-34]. In doing so, these advances have greatly improved the design of intelligent receivers and paved the way for new research areas for future wireless communications. Furthermore, recently, transformer-based wireless communication models and hardware impairment-aware learning methods have shown great potential in wireless signal processing with the challenge of computational complexity and deployment for large-scale massive MIMO systems [35-37]. Although significant progress has been made in these areas, most of the previous works have concentrated on enhancing the detection algorithm or the network architecture, and very few works have attempted to solve multiple hardware impairments in a single hybrid receiver. In addition, the computational complexity of many deep unfolding and graph-based methods becomes significant as the system size grows, making them hard to apply to practically large-scale MIMO systems. Inspired by these challenges, a lightweight AI-assisted hybrid receiver is proposed in this work, which brings into the hybrid receiver the combination of MMSE detection, AI-based residual error correction, and adaptive fusion to achieve a good performance-complexity trade-off, while enhancing the resistance to hardware impairments. There have been some studies on developing an AI-aided MIMO detector or a deep learning receiver in the model, but most works have tackled only one of the two aspects at a time. The proposed framework, however, provides a unified hybrid receiver, which combines a lightweight module for post-detection error correction based on AI technology and an adaptive fusion strategy to enhance the MMSE output in the presence of multiple simultaneous hardware impairments. The proposed receiver simultaneously takes into account the phase noise, IQ imbalance, nonlinearity of the PA, and imperfect CSI in a single detection framework, in contrast to previous approaches that were designed for ideal hardware or a single impairment. Furthermore, the adaptive fusion mechanism dynamically adjusts the weightage of the conventional detector and AI module, achieving better bit error rate (BER) performance and robustness with a moderate computational complexity. These are the main characteristics that make the proposed receiver different from the existing AI-assisted hybrid detection approaches and represent the primary methodological contribution of this work. To offer an intuitive grasp of this restriction and the suggested solution, Figure 1 demonstrates the impact of hardware limits on the traditional receivers and the improvement gained by the proposed hybrid AI-assisted design. The conventional receiver just employs linear detection, whereas the proposed design improves the detection stage with AI-based correction and adaptive fusion. As shown in Figure 1, this conceptual graphic attempt to show the broad processing flow and the essential elements of the proposed method before providing the detailed system framework and mathematical formulation. This paper proposes a new hybrid AI-assisted receiver for hardware-constrained large MIMO systems. The suggested technique combines a traditional MMSE detector with an extremely lightweight AI-based error correction module. An adaptive fusion approach is then applied to accurately combine the outcomes of both stages. This method may efficiently decrease the stray interference and the nonlinear distortions with a relatively low processing complexity.
Figure 1. Conceptual schematic of the proposed conventional and hybrid receiver systems with hardware deficits
The main contributions of this work are briefly described as follows:
•Regarding massive MIMO systems with realistic hardware constraints, a hybrid receiver architecture with AI support is recommended.
•A convenient AI-based error-correcting module is intended to outperform conventional linear processing in terms of detection accuracy.
•An adaptive fusion mechanism is proposed to optimally combine the linear and AI-based outputs for resilience.
•The performance is thoroughly evaluated in the presence of several impairment scenarios, including imperfect CSI, phase noise, IQ imbalance, and PA nonlinearity.
•The proposed technique is shown to provide a favorable performance–complexity trade-off, significant BER improvement and better spectrum efficiency over conventional detectors.
The rest of this paper is structured as follows. The system model is presented in Section 2. The suggested hybrid receiver is explained in Section 3. The performance evaluation and simulation results are covered in Section 4. The article is finally concluded in Section 5.
The significant MIMO system model with realistic hardware impairments is described in this section. Figure 2 shows the proposed receiver's whole signal processing chain. Consider an uplink massive MIMO system with K single-antenna users broadcasting concurrently to a base station with Nr antennas, where Nr > K [5-7]. Each member in the transmitted symbol vector, represented as × ∈ CK × 1, represents a modulated symbol (such as QPSK or 16-QAM) [13].
Figure 2. System model of the proposed hybrid AI-assisted massive multiple-input multiple-output (MIMO) receiver
2.1 Hardware-impaired transmitter model
PA nonlinearity affects the transmitted signal in practical systems. Amplitude-to-phase (AM/PM) and amplitude-to-amplitude (AM/AM) characteristics can be used to characterize this distortion. The transmitted signal can thus be written as [8, 16]: $\widetilde{\mathrm{x}}$ = f(x), where f(⋅) is the nonlinear transformation that the PA introduced. A Rayleigh fading channel, H ∈ CNr × K, is used to describe the wireless channel between the base station and users. Each element of this channel has a complicated Gaussian distribution [5, 6]. Additive white Gaussian noise (AWGN), represented by n∼CN (0, σ2I), further taints the received signal.
2.2 Impaired receiver hardware
Phase noise and IQ imbalance are two further limitations that are taken into account at the receiver side. These deficiencies cause distortions that are capable of being represented by additive and multiplicative components [10, 16] can be used to indicate the total received signal:
$y=D_r H D_t x+n$ (1)
where, the transmitter and receiver impairment matrices are denoted by Dt and Dr, respectively.
Perfect CSI is unavailable in real-world scenarios. Rather, the receiver uses an approximated channel matrix $\widehat{H}$, which is described as follows [10, 11]:
$\widehat{H}=\sqrt{1-\tau} H+\tau E$ (2)
where, E is the estimated noise matrix and τ stands for the degree of estimation inaccuracy. The estimated channel is used for conventional linear detection. The MMSE detector, for instance, is provided by [12, 13]:
$\widehat{\mathrm{x}}_{\mathrm{MMSE}}=\left(\widehat{\mathrm{H}}^{\mathrm{H}} \widehat{\mathrm{H}}+\sigma^2 \mathrm{I}\right)^{-1} \widehat{\mathrm{H}}^{\mathrm{H}} \mathrm{y}$ (3)
Although MMSE beats MRC and ZF, it is still influenced by hardware impairments and CSI errors.
This section presents the proposed hybrid receiver design, which aims to enhance detection performance in massive MIMO systems under hardware constraints. The proposed approach combines conventional linear detection, an AI-based error correction module, and an adaptive fusion approach.
3.1 Linear minimum mean square error detection
A conventional linear detector is initially used to assess the incoming signal based on the expected channel matrix $\widehat{\mathrm{H}}$. Among linear detectors, the MMSE technique provides a beneficial trade-off between noise reduction and interference mitigation. Eq. (3) provides an MMSE estimation of the transmitted signal. An AI-based error correction module is used to make up for the residual mistakes caused by defective CSI and hardware limitations [12, 20]. The AI module functions as a post-processing step that improves the MMSE output rather than taking the place of the linear detector. The first estimate is represented by $\hat{x}_{M M S E}$. A nonlinear mapping is learned by the AI module.
$\hat{x}_{A I}=\mathcal{F}_A\left(\hat{x}_{M M S E}\right)$ (4)
A lightweight neural network specified by θ is represented as $\mathcal{F}_\theta(\cdot)$ [12, 25]. The objective of the AI module is to compensate for residual detection error by detecting nonlinear distortions due to PA nonlinearity, phase noise, IQ imbalance, CSI imperfection. The perfect design is intentionally selected [20, 22] to ensure low computing complexity and practical execution.
3.2 AI-based error correction module
After detection of MMSE, residual errors due to practical hardware impairments (such as phase noise, IQ imbalance, PA nonlinearity, and inaccurate CSI) remain. As a post-detection error refinement stage, a lightweight AI-based error correction module is added to further suppress these nonlinear residual errors. The neural network is not used to replace the conventional detector but rather performs an error-learning operation of the residual error pattern and provides corrected symbol estimates before the adaptive fusion stage. This design provides an increased detection accuracy with reasonable computational complexity. For further illustration of the internal structure of the proposed error correction module based on an AI system, the entire processing flow of the lightweight NN used after the MMSE detection is given by Figure 3. The network is to approximate and mitigate the remaining nonlinear errors caused by the practical hardware distortion in front of the adaptive fusion stage.
Figure 3. The suggested lightweight AI-based error-correcting module's architecture
The proposed AI correction module is a simple feed-forward neural network following the MMSE detector, as depicted in Figure 3. The detected symbols are normalized and then passed through a series of hidden layers to approximate the residual nonlinear errors that occur due to the practical hardware impairments. In the end, the edited symbols are passed to the adaptive fusion block, which then fuses the output from the MMSE detector with the corrected symbols to increase the detection robustness. The configuration of the proposed neural network is described in detail in Table 1. To achieve such a balance between the accuracy of detection and the complexity of computation, the architecture chosen is suitable for practical massive MIMO receivers.
Table 1. The suggested lightweight AI-based error-correcting module configuration
|
Parameter |
Configuration |
|
Network Type |
Feedforward Neural Network (MLP) |
|
Input Features |
4 |
|
Hidden Layers |
2 |
|
Hidden Layer 1 |
64 neurons (ReLU) |
|
Hidden Layer 2 |
32 neurons (ReLU) |
|
Output Layer |
2 neurons (Linear) |
|
Loss Function |
Mean Squared Error (MSE) |
|
Optimizer |
Adam |
|
Learning Rate |
0.001 |
|
Batch Size |
128 |
|
Epochs |
50 |
|
Training Dataset |
Monte Carlo generated samples |
|
Training Samples |
100,000 |
|
Deployment |
One trained model used during testing |
The designed neural network is designed to be lightweight and to have as few computational resources as possible to achieve high detection accuracy. The training data consists of Monte Carlo generated data for various hardware impairments and SNR levels, and the model learns the nonlinear residual distortion after the MMSE detector. The same model is used for all simulation scenarios, without the need to retrain for each channel realization, reducing implementation complexity while maintaining robust performance.
3.3 Mechanism of adaptive fusion
The AI-based correction increases detection accuracy, but channel circumstances can affect how well it works. Consequently, the outputs of the AI module and the MMSE detector are combined using an adaptive fusion technique. The final signal that is detected is as follows [21, 25]:
$\hat{x}=\omega \hat{x}_{A L}+(1-\omega) \hat{x}_{M M S E}$ (5)
where, the adaptive weighting factor is represented by ω ∈ [0,1]. Depending on the following system circumstances, the fusion weight ω may be considered dynamically changed estimated SNR, channel quality, and error statistics.
IQ imbalance is regarded as one of the main hardware flaws at the receiver, along with phase noise. The received signal is distorted in practical transceivers due to amplitude and phase errors caused by mismatches between the in-phase (I) and quadrature (Q) branches. [8, 16] can be used to mimic this effect:
$y_O=\alpha y+j \beta y^*$ (6)
where, * denotes the complex conjugate, y represents a perfect received signal, and α and β are parameters that represent the extent of the IQ imbalance. The leakage between the I and Q components, which creates self-interference and reduces detection accuracy, is captured by this model. This degradation becomes more severe, necessitating robust receiver design because there are many RF chains in huge MIMO systems [9, 17]. The suggested hybrid receiver uses adaptive fusion and AI-based error correction to reduce these distortions. The purpose of the suggested hybrid receiver is to attain better performance with a low computational overhead. The AI module adds additional calculations based on the network size, whereas the MMSE step includes matrix inversion with complexity O(K3). The suggested approach strikes a good balance between detection accuracy and computing cost, as shown in the performance–complexity study. The following is a summary of the suggested hybrid receiver. The following is a summary of the suggested hybrid receiver:
The receiver hierarchical architecture effectively reduces nonlinear distortions and residual interference, potentially improving performance under actual system impairments.
3.4 Computational complexity analysis
The theoretical analysis of the computational complexity is performed for each processing stage to further validate the efficiency of the proposed hybrid receiver. Theoretical complexity measures the computational cost and unlike run-time measurements it is independent of the implementation platform and hardware. The analysis focuses on the conventional detectors and the proposed AI-based error correction module, the adaptive fusion stage and the overall receiver architecture. Theoretical computational complexity for the considered receivers is summarized in Table 2, where Nr and K are the number of receive antennas and users, respectively, and NAI is the computational complexity of the lightweight neural network.
Table 2. Receivers' theoretical computational complexity
|
Method |
Computational Complexity |
|
Proposed Receiver |
O(K3+NrK2+NAI) |
|
Adaptive Fusion |
O(K) |
|
AI Module |
O(NinN1+N1N2+N2Nout) |
|
MMSE |
O(K3+NrK2) |
|
ZF |
O(K3+NrK2) |
|
MRC |
O(NrK) |
Table 2 shows that the MMSE detection block accounts for the majority of the computational burden of the suggested receiver, with the lightweight AI module and adaptive fusion adding very little computational cost. According to the simulation findings, this makes the suggested system appropriate from the perspective of detection performance with a modest performance complexity trade-off.
This section presents a comprehensive performance evaluation of the proposed hybrid AI-assisted receiver for massive MIMO systems under actual conditions. The study looks at computational complexity, hardware constraints, incorrect CSI and scalability in terms of number of users. The baseline is given by conventional linear detectors such as MRC, ZF and MMSE.
4.1 Simulation setup
This section provides details about the simulation environment and the system configuration that is applied to test the proposed hybrid receiver in real-world wireless communications scenarios. To make a fair and meaningful performance assessment, the performance was evaluated using both the conventional receiver and the AI-assisted receiver under the same simulation environment. In the simulation setup, typical hardware impairments such as phase noise, IQ imbalance, PA nonlinearity and imperfect CSI were included to represent realistic massive MIMO communication scenarios. Furthermore, all the parameters used in the simulation were kept the same for all the experiments to ensure reliability and repeatability of the results obtained. The principal simulation parameters used throughout this work are listed in Table 3 to ensure the framework proposed is reproducible and to give a complete description of the simulation environment.
Table 3. Simulation settings used for the performance assessment
|
Parameter |
Value |
|
Platform Simulation |
MATLAB R2020b |
|
Trials of Monte Carlo |
10⁵ |
|
MIMO Configuration |
64 × 16 |
|
Modulation |
QPSK (experiments selected 16-QAM) |
|
Channel Model |
Rayleigh + AWGN |
|
SNR Range |
0–30 dB |
|
CSI Condition |
Perfect/Imperfect |
|
Impairments Hardware |
PA Nonlinearity, IQ Imbalance, Phase Noise |
|
Processor |
Intel Core i7 |
|
Memory |
16 GB RAM |
|
Operating System |
Windows 10 |
All simulation results were carried out with the same system configuration, which is shown in Table 3, to make them comparable and provide a fair comparison among the considered receivers. The simulation configuration comprises of the MIMO antenna configuration, the modulation scheme, the channel model, the simulation hardware impairments, the number of Monte Carlo trials, the range of SNR points, and the software/hardware environment used for the performance evaluation.
4.2 Hardware impairments and bit error rate performance
The BER performance as a function of SNR with combined hardware impairments phase noise (PN = 6°), IQ imbalance (IQ = 5°), PA nonlinearity and defective CSI ($\tau$ = 0.15) is illustrated in Figure 4. As one would expect, the accumulation of distortions leads to a significant performance degradation in traditional detectors. In particular, the MRC detector shows almost flat performance across the SNR range, with a BER of about 10-1 even for high SNR values, indicating that it is not able to effectively mitigate the hardware-induced interference and distortions. While ZF and MMSE detectors outperform the others, their BER curves trend to saturation at high SNRs, emphasizing the limitations of linear processing in the presence of impairments. ZF and MMSE achieve BER values of about 2 × 10-2 and 1.5 × 10-2 at SNR 30 dB, respectively. On the contrary, the proposed hybrid receiver demonstrates a considerable performance improvement at all SNR values. In particular, the proposed technique achieves a BER of roughly 3 × 10-3 at SNR 30 dB, which is more than one order of magnitude better than ZF and more than two orders better than MMSE. This gain shows the effectiveness of integrating adaptive fusion with AI-based error correction, enabling the receiver to compensate for residual interference and nonlinear distortions, which dominate in high-SNR regimes.
Figure 4. Bit error rate (BER) performance comparison of maximum ratio combining (MRC), zero-forcing (ZF), minimal mean square error (MMSE) and suggested hybrid receiver with perfect channel state information (CSI)
4.3 Spectral efficiency with hardware impairments
The spectral efficiency performance of the suggested receiver under significant hardware impairments is shown in Figure 5, such as PA nonlinearity, phase noise, IQ imbalance, and CSI uncertainty. When compared to MRC, ZF, and MMSE detectors, it is evident that the suggested hybrid receiver continuously obtains the maximum spectral efficiency over the whole SNR range. Conventional detectors often saturate at lower throughput levels, whereas this benefit becomes more noticeable at medium and high SNR values. The MRC detector, in particular, reaches the lowest spectral efficiency, saturating at about 11 bits/s/Hz. With increases of about 11.5 and 12.3 bits/s/Hz at high SNR levels, respectively, the ZF and MMSE detectors offer moderate improvements. In contrast, over the whole SNR range, the suggested hybrid receiver performs noticeably better than all baseline techniques. The suggested approach provides a spectrum efficiency of about 16.1 bits/s/Hz at SNR = 30 dB, which is about double that of MMSE. Additionally, the performance improvement is especially noticeable in the mid-to-high SNR range (10–20 dB), where the suggested technique climbs quickly before stabilizing, demonstrating effective channel capacity use even in the face of hardware defects. This significant improvement shows how well the suggested hybrid architecture mitigates nonlinear distortions while simultaneously improving throughput and dependability.
The outcome demonstrates that under realistic non-ideal operating conditions, the suggested receiver enhances both total system throughput and dependability. For an instance user in the massive MIMO system under discussion, Figure 6 shows the distortion and interference impacts before and after receiver processing to give a clear visual and quantitative knowledge of the impact of hardware impairments. The graphic illustrates how the suggested hybrid receiver greatly enhances signal quality, whereas traditional linear detection only somewhat mitigates these impacts. Noise and hardware flaws cause the distorted signal to spread significantly. Four distinct phases of signal processing are displayed in Figure 6. The degraded received signal is displayed before to any processing in Figure 6(a). It is clear that considerable distortion and interference are introduced by hardware defects, leading to widely dispersed constellation points that depart from their optimal placements. The result of the traditional MMSE detector is shown in Figure 6(b). The limits of linear detection under hardware impairments are demonstrated by the substantial residual interference that persists even though MMSE improves clustering and lowers some of the distortion. The output of the suggested hybrid receiver is shown in Figure 6(c).
Figure 5. Spectral efficiency against signal-to-noise ratio (SNR) for various detectors under hardware limitations, demonstrating the hybrid receiver's improved performance
The constellation points densely cluster around their optimal locations, indicating how well the AI-based correction and adaptive fusion work to reduce interference and distortion. Additionally, a quantitative comparison of the residual error power is shown in Figure 6(d). After MMSE detection, the residual error decreases by around 57.8%, from roughly 1.29 × 10⁻¹ before to processing to 5.47 × 10⁻². In comparison to the communication scenario, the error is further decreased to 5.17 × 10⁻³ with the suggested hybrid receiver, reaching about 96% reduction. This significant augmentation, which is almost an order of magnitude more than MMSE, validates the effectiveness of the recommended technique.
4.4 Effects of inadequate channel state information
The performance under different CSI imperfection levels is shown in Figure 7. It is found that when CSI error increases, all detection techniques gradually deteriorate as a result of imprecise channel estimates. With BER rising from around 2 × 10-2 at τ = 0.05 to roughly 3.5 × 10-2 at τ = 0.5, the MRC detector shows the strongest sensitivity to CSI errors. At high CSI error levels, ZF and MMSE detectors also exhibit considerable deterioration, with BER values of around 1.8 × 10-2 and 2.5 × 10-2, respectively. Notably, the suggested hybrid receiver shows far better resilience to CSI flaws. The suggested approach produces a BER of about 5 × 10-3 at low error levels (τ = 0.05), which is greater than an order of magnitude lower than MMSE. Even at τ = 0.5, the BER stays below 3 × 10-2 while increasing with larger CSI error. However, for all CSI error levels, the suggested hybrid receiver continuously performs better than traditional methods. The AI-assisted correction stage, which successfully corrects for channel estimation errors and improves detection robustness beyond what is possible with just linear detectors, is responsible for this improvement. The trade-off between BER performance and computational complexity is examined in Figure 8. Pareto optimality, runtime comparison, BER than the AI-only approach while just slightly increasing computing cost when compared to MMSE. The BER comparison is shown in Figure 8(c), which shows that the suggested strategy outperforms MMSE by more than an order of magnitude and the AI-only approach by about 25 times.
Figure 6. Distortion and residual error visualization in a 64 × 16 massive multiple-input multiple-output (MIMO) system before to and during receiver processing: (a) attenuated received signal; (b) minimal mean square error (MMSE) detection; (c) suggested hybrid receiver; and (d) residual error power comparison
Lastly, the Pareto optimality analysis is shown in Figure 8(d). All the techniques taken into consideration, the suggested hybrid receiver achieves the greatest balance between performance and computational complexity since it is closer to the ideal area (low BER and low runtime). When compared to MMSE, the suggested technique reduces BER by around 85–90% while needing only about 25% more runtime. The suggested hybrid receiver achieves a far greater performance improvement than the MMSE detector, despite a little runtime increase. The Pareto analysis clearly indicates that the recommended strategy provides the least BER with a slight computational price and works in a near-optimum area. The simulation results prove that the proposed receiver is a promising solution for the massive MIMO system. These results confirm the capability of the proposed hybrid architecture to combine the advantages of the AI-based correction and of the linear detection, achieving considerable performance increases without incurring excessive processing overhead. For a clearer quantification of the performance-complexity trade-off, Table 4 provides a comprehensive comparison of the receivers under discussion in terms of BER, runtime, computational complexity, robustness and scalability under hardware impairments.
Figure 7. Bit error rate (BER) performance vs channel state information (CSI) error level, demonstrating the hybrid receiver's resilience to inaccurate channel estimates
Figure 8. Bit error rate (BER)–runtime trade-off, runtime comparison, BER comparison, and Pareto optimality analysis are instances of performance–complexity trade-offs
Table 4. Hardware impairment-listed performance comparison of several receivers
|
Method |
BER @ 20 dB |
Runtime (ms) |
Complexity |
Robustness |
Scalability |
|
MRC |
1.5e-2 |
1.2 |
Low |
Low |
Poor |
|
ZF |
6.0e-3 |
1.5 |
Medium |
Medium |
Moderate |
|
MMSE |
1.2e-4 |
1.8 |
Medium |
Good |
Good |
|
AI-only |
4.0e-5 |
2.6 |
High |
Good |
Good |
|
Proposed Hybrid |
1.0e-6 |
2.1 |
Medium |
Excellent |
Excellent |
Table 4 provides a quantitative comparison of the receivers under examination in terms of the BER, runtime, computational complexity, resilience and scalability under hardware impairments. It can be demonstrated that MRC is the worst with a BER of approximately 1.5 × 10-2, ZF is a substantial improvement but its complexity is just slightly higher than MRC. The MMSE detector considerably increases the performance with modest complexity and BER of around 1.2 × 10-4. The AI-only solution further reduces the BER to roughly 4 × 10-5, at the expense of increased runtime of 2.6 ms. The proposed hybrid receiver can outperform MMSE by nearly two orders of magnitude with a BER of around 1 × 10-6 and a runtime of 2.1 ms. Furthermore, the suggested technique has excellent scalability and robustness, and shows its effectiveness under practical system conditions.
4.5 Ablation study
An ablation investigation is shown in Figure 9 to further confirm the design of the suggested architecture. Starting with the classic MMSE detector, it is clear that there is minimal boost in performance, with BER saturating at roughly 10-4 at high SNR. In order progressively add the AI-based error correcting module, there is a clear reduction in BER. The detection accuracy is considerably improved by adding the AI-based correction module (MMSE + AI), where the BER is reduced to roughly 7 × 10-5 at SNR = 30 dB. Adaptive fusion (MMSE + Fusion) enhances the robustness under various channel conditions and mitigates the BER to around 8 × 10-5. The full hybrid receiver works well at all SNR levels with BER close to 1 × 10−6 at SNR = 30 dB. The results show a clear decrease in the BER with the inclusion of more components. In particular, the proposed hybrid technique improves the MMSE by nearly two orders of magnitude at high SNR. This trend illustrates that each module (AI correction, adaptive fusion) helps successfully in minimizing residual errors due to hardware imperfections, thereby providing a considerable improvement in overall performance. The consistent improvement illustrates the synergy of linear detection, AI-based enhancement and adaptive fusion to mitigate nonlinear distortions.
Figure 9. An ablation study demonstrates the integration of AI and fusion stages in the proposed hybrid receiver to improve performance
The scalability of the receivers considered in the multi-user massive MIMO systems is shown in Figure 10, which shows the BER performance vs the number of users. The system suffers increased interference with the rise in users and all detectors performance degrades. The worst performing detector is the MRC detector with a BER close to 10-1 in all users configurations proving its failure to sufficiently suppress interference. The BER of the ZF and MMSE detectors increases from around 1 × 10-3 at K = 4 to about 8 × 10-2 and 9.3 × 10-2, respectively, at K = 64, indicating reasonable performance. In contrast, the suggested hybrid receiver consistently performs better than all baseline techniques. It achieves a BER of about 7 × 10-4 at low user density (K = 4) and stays below 4 × 10-2 at high user density (K = 64). At large user loads, this is almost a one-order-of-magnitude improvement over MMSE. In spite of this, the proposed hybrid receiver continues to perform better in every user setup. The BER attained by the suggested approach is still much lower than that of MMSE even at large user densities, indicating its robustness against multi-user interference and good scalability. The results show that the proposed hybrid receiver is reliable with increasing system demand and hence, is very suitable for dense multi-user scenarios.
Figure 10. Scalability analysis showing the resilience of the proposed hybrid receiver under growing system load in terms of bit error rate (BER) vs user count
In this study, a hybrid AI-assisted receiver was presented for huge MIMO systems with actual hardware impairments, including phase noise, IQ imbalance, PA nonlinearity and insufficient CSI. The suggested solution combines linear MMSE detection with a lightweight AI-based correction module and adaptive fusion for improved detection robustness. The simulation findings showed that the traditional linear detectors such as MRC, ZF and MMSE degrade significantly by the hardware impairments, especially in the high SNR regime where the BER saturates. On the other hand, the suggested hybrid receiver achieves a significant performance improvement. For example, with SNR = 25 dB, the BER is decreased from around 10⁻3 (proposed hybrid) to less than close to 10⁻6, which is close to one order of magnitude improvement.
Furthermore, the spectrum efficiency findings reveal that the suggested approach outperforms substantially the traditional detectors, providing roughly 16 – 17 bits/sec/Hz vs 11 – 12 bits/sec/Hz for MMSE, thereby doubling the possible rate under the severe impairment circumstances. The robustness of the suggested technique was further verified under imperfect CSI scenarios, where the hybrid receiver maintained steady performance while the traditional detectors showed obvious degradation with the rise of CSI error. Additionally, the ablation investigation verified that each component of the suggested architecture contributes to the overall performance enhancement, and the whole hybrid design achieved the lowest BER compared to MMSE, MMSE+AI, and MMSE+fusion configurations. The suggested technique provides a good performance-complexity trade-off in terms of complexity. It achieves a far lower BER than MMSE, at the cost of just modest more processing, and is still much more efficient than completely AI-based techniques.
Finally, the results for different number of users reveal that the proposed hybrid receiver scales well with the system size and achieves higher performance with increasing numbers of users. The trade-off between the performance and complexity analysis results show that the proposed technique is able to provide an effective trade-off between the detection performance and computational complexity in the simulation scenarios considered in this paper. Overall, within practical hardware limitations, the suggested hybrid receiver offers a promising option for dependable and effective communication in 6G massive MIMO systems. Future research will concentrate on applying the suggested framework to real-time hardware platforms and expanding it to more complicated channel models, such as correlated fading and mobility situations.
[1] Saad, W., Bennis, M., Chen, M. (2019). A vision of 6G wireless systems: Applications, trends, technologies, and open research problems. IEEE Network, 34(3): 134-142. https://doi.org/10.1109/MNET.001.1900287
[2] You, X., Wang, C.X., Huang, J., et al. (2021). Towards 6G wireless communication networks: Vision, enabling technologies, and new paradigm shifts. Science China Information Sciences, 64(1): 110301. https://doi.org/10.1007/s11432-020-2955-6
[3] Dai, L., Wang, B., Peng, M., Chen, S. (2018). Hybrid precoding-based millimeter-wave massive MIMO-NOMA with simultaneous wireless information and power transfer. IEEE Journal on Selected Areas in Communications, 37(1): 131-141. https://doi.org/10.1109/JSAC.2018.2872364
[4] Yu, W., He, H., Song, S., et al. (2025). AI and deep learning for terahertz ultra-massive MIMO: From model-driven approaches to foundation models. Engineering. https://doi.org/10.1016/j.eng.2025.07.032
[5] Björnson, E., Sanguinetti, L. (2019). Making cell-free massive MIMO competitive with MMSE processing and centralized implementation. IEEE Transactions on Wireless Communications, 19(1): 77-90. https://doi.org/10.1109/TWC.2019.2941478
[6] Björnson, E., Sanguinetti, L. (2020). Scalable cell-free massive MIMO systems. IEEE Transactions on Communications, 68(7): 4247-4261. https://doi.org/10.1109/TCOMM.2020.2987311
[7] Ngo, H.Q. (2020). Cell-free massive MIMO. In Encyclopedia of Wireless Networks, Cham: Springer International Publishing, pp. 165-169. https://doi.org/10.1007/978-3-319-78262-1_137
[8] Demir, Ö.T., Björnson, E. (2019). Channel estimation in massive MIMO under hardware non-linearities: Bayesian methods versus deep learning. IEEE Open Journal of the Communications Society, 1: 109-124. https://doi.org/10.1109/OJCOMS.2019.2959913
[9] Björnson, E., Hoydis, J., Kountouris, M., Debbah, M. (2014). Massive MIMO systems with non-ideal hardware: Energy efficiency, estimation, and capacity limits. IEEE Transactions on Information Theory, 60(11): 7112-7139. https://doi.org/10.1109/TIT.2014.2354403
[10] Li, Q., El-Hajjar, M., Sun, Y., et al. (2023). Achievable rate analysis of the STAR-RIS-aided NOMA uplink in the face of imperfect CSI and hardware impairments. IEEE Transactions on Communications, 71(10): 6100-6114. https://doi.org/10.1109/TCOMM.2023.3287995
[11] Xie, M., Yu, X., Wang, K., Zhang, J., Dang, X., Yuen, C. (2024). Superimposed pilots for cell-free massive MIMO over spatial-correlated Rician fading channels. IEEE Transactions on Wireless Communications, 23(12): 19537-19552. https://doi.org/10.1109/TWC.2024.3485070
[12] He, H., Wen, C.K., Jin, S., Li, G.Y. (2020). Model-driven deep learning for MIMO detection. IEEE Transactions on Signal Processing, 68: 1702-1715. https://doi.org/10.1109/TSP.2020.2976585
[13] Albreem, M.A., Juntti, M., Shahabuddin, S. (2019). Massive MIMO detection techniques: A survey. IEEE Communications Surveys & Tutorials, 21(4): 3109-3132. https://doi.org/10.1109/COMST.2019.2935810
[14] Azizi, S.P., Cheng, S.C., Lu, H.Y. (2023). Deep learning detection for massive MIMO systems. In 2023 22nd International Symposium on Communications and Information Technologies (ISCIT), Sydney, Australia, pp. 1-6. https://doi.org/10.1109/ISCIT57293.2023.10376038
[15] Ye, H., Li, G.Y., Juang, B.H. (2017). Power of deep learning for channel estimation and signal detection in OFDM systems. IEEE Wireless Communications Letters, 7(1): 114-117. https://doi.org/10.1109/LWC.2017.2757490
[16] Marzetta, T.L. (2010). Noncooperative cellular wireless with unlimited numbers of base station antennas. IEEE Transactions on Wireless Communications, 9(11): 3590-3600. https://doi.org/10.1109/TWC.2010.092810.091092
[17] Ngo, H.Q., Larsson, E.G., Marzetta, T.L. (2013). Energy and spectral efficiency of very large multiuser MIMO systems. IEEE Transactions on Communications, 61(4): 1436-1449. https://doi.org/10.1109/TCOMM.2013.020413.110848
[18] Larsson, E.G., Edfors, O., Tufvesson, F., Marzetta, T.L. (2014). Massive MIMO for next generation wireless systems. IEEE Communications Magazine, 52(2): 186-195. https://doi.org/10.1109/MCOM.2014.6736761
[19] Hassibi, B., Vikalo, H. (2005). On the sphere-decoding algorithm I. Expected complexity. IEEE Transactions on Signal Processing, 53(8): 2806-2818. https://doi.org/10.1109/TSP.2005.850352
[20] O’Shea, T., Hoydis, J. (2017). An introduction to deep learning for the physical layer. IEEE Transactions on Cognitive Communications and Networking, 3(4): 563-575. https://doi.org/10.1109/TCCN.2017.2758370
[21] Tebe, P.I., Wen, G., Li, J., Huang, Y., Ampoma, A.E., Gyasi, K.O. (2019). Massive MIMO with transceiver hardware impairments: Performance analysis and phase noise error minimization. KSII Transactions on Internet and Information Systems, 13(5): 2357-2380. https://doi.org/10.3837/tiis.2019.05.007
[22] Demir, Ö.T., Björnson, E. (2025). Cell-free massive MIMO with hardware-impaired wireless fronthaul. In 2025 59th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA, pp. 1656-1660. https://doi.org/10.1109/IEEECONF67917.2025.11443864
[23] Ali, A., Pourrostam, J. (2025). A survey on cell-free massive MIMO system with hardware impairments. Physical Communication, 72: 102743. https://doi.org/10.1016/j.phycom.2025.102743
[24] Samuel, N., Diskin, T., Wiesel, A. (2019). Learning to detect. IEEE Transactions on Signal Processing, 67(10): 2554-2564. https://doi.org/10.1109/TSP.2019.2899805
[25] Doha, S.R., Abdelhadi, A. (2025). Deep learning in wireless communication receivers: A survey. IEEE Access. https://doi.org/10.1109/ACCESS.2025.3584000
[26] Yu, W., Sohrabi, F., Jiang, T. (2022). Role of deep learning in wireless communications. IEEE BITS the Information Theory Magazine, 2(2): 56-72. https://doi.org/10.1109/MBITS.2022.3212978
[27] Ibarra-Hernández, R.F., Castillo-Soria, F.R., Gutiérrez, C.A., et al. (2025). Efficient deep learning-based detection scheme for MIMO communication systems. Sensors, 25(3): 669. https://doi.org/10.3390/s25030669
[28] Eisen, M., Ribeiro, A. (2019). Optimal wireless resource allocation with random edge graph neural networks. IEEE Transactions on Signal Processing, 68: 2977-2991. https://doi.org/10.1109/TSP.2020.2988255
[29] Zappone, A., Di Renzo, M., Debbah, M. (2019). Wireless networks design in the era of deep learning: Model-based, AI-based, or both? IEEE Transactions on Communications, 67(10): 7331-7376. https://doi.org/10.1109/TCOMM.2019.2924010
[30] Liao, J., Zhao, J., Gao, F., Li, G.Y. (2020). A model-driven deep learning method for massive MIMO detection. IEEE Communications Letters, 24(8): 1724-1728. https://doi.org/10.1109/LCOMM.2020.2989672
[31] Hu, Q., Cai, Y., Zhang, G., Yu, G., Li, G.Y. (2023). Deep-unfolding for next-generation transceivers. arXiv preprint arXiv:2305.08303. https://doi.org/10.48550/arXiv.2305.08303
[32] Zhu, J., Chang, T.H., Xiang, L., Shen, K. (2026). DeepFP: Deep-unfolded fractional programming for MIMO beamforming. IEEE Transactions on Communications. https://doi.org/10.1109/TCOMM.2026.3653777
[33] Xu, Q., Sun, J., Xu, Z. (2025). Efficient MU-MIMO beamforming based on majorization-minimization and deep unfolding. IEEE Transactions on Wireless Communications, 24(5): 3949-3963. https://doi.org/10.1109/TWC.2025.3541144
[34] Xu, B., Zhang, J., Lin, Q., Xiao, H., Wu, Y.C., Ai, B. (2024). Deep unfolding beamforming and power control designs for multi-port matching networks. IEEE Transactions on Wireless Communications, 24(2): 1401-1414. https://doi.org/10.1109/TWC.2024.3509276
[35] Xu, H., Liang, Y., Xie, R., Kong, Y. (2026). The deep learning evolution in wireless physical layer communications: Applications, challenges, and evolutionary directions. Sensors, 26(11): 3609. https://doi.org/10.3390/s26113609
[36] Yu, H., Vo, C.H., Lee, C. (2026). Detection-guided deep unfolding for joint underwater image enhancement and object detection. IEEE Access, 14: 17743-17759. https://doi.org/10.1109/ACCESS.2026.3659134
[37] Chowdhury, A., Verma, G., Swami, A., Segarra, S. (2023). Deep graph unfolding for beamforming in MU-MIMO interference networks. IEEE Transactions on Wireless Communications, 23(5): 4889-4903. https://doi.org/10.1109/TWC.2023.3323207
