Signal Processing Algorithms for Mean Square Error Analysis in MIMO Wireless Transceivers

Signal processing is an integral component in the chain of information transfer for wireless transceivers. This field classifies signals by dimensionality: one-dimensional (1D) signals such as speech, two-dimensional (2D) signals like images, and three-dimensional (3D) signals, which include video. These signals, when contaminated by additive white Gaussian noise (AWGN) within a wireless transceiver, necessitate sophisticated processing algorithms for accurate information retrieval. Irrespective of the transceiver's system model or design specifications, the use of pilot signals [1] is essential for the efficient transfer of information. Pilot signals, which act as known reference signals for transmitters and receivers, enable the accurate processing of transmitted signals and the determination of mean square error (MSE) [2]. This determination is critical for managing data transmission and reception, whether through a single antenna system or a multiple input multiple output (MIMO) wireless transceiver [3]. MIMO systems, in particular, are known for their potential to provide increased diversity gain, spectral efficiency, reliability, and coverage in multipath propagation contexts [4].

Signal processing in MIMO systems [5] is considered at various levels, from architectural design in the radio frequency (RF) and in baseband stages. At the baseband level, digital modulation, such as orthogonal frequency division multiplexing (OFDM), is prominent in current 5G technology and anticipated 6G developments utilizing massive MIMO methodologies [6, 7]. The employment of pilot signals, constructed from discrete Fourier transform (DFT) matrices [8], facilitates the estimation of broadcast channel conditions through MSE when transmitted across wireless channels.

MSE is further attributed to it statistical components mean and variance. Here, the mean is the first moment of a random signal, and variance represents the power level of a signal corrupted by noise. Signal processing algorithms, including least squares (LS), minimum mean square error (MMSE), and maximum likelihood (ML) approaches [9], are utilized to estimate pilot signals within MIMO wireless transceivers. The LS algorithm minimizes error at the signal level by employing an optimization function. In contrast, MMSE and ML approaches rely on conditional probability density functions and likelihood functions for parameter estimation.

At the transmitter end, signal processing begins with the conversion of information from analog to digital form. This digital data undergoes source coding through Huffman and Shannon-Fano coding to minimize redundancy, followed by channel coding for error control [10]. Precoding schemes are also integrated at the MIMO transmitter to facilitate data transfer over wireless channels. The receiver end employs decoding and Bayesian detection to recover digital data signals [11], and ML based algorithms are used to minimize error in stationary Gaussian sources [12].

The research article [13] explores applications of MSE in advanced communication scenarios, including MIMO millimeter-wave communications, reconfigurable intelligent surface (RIS) enhanced multiuser MIMO systems in [14], and beamforming optimization [15]. In the network layer aspect of 5G systems, MSE analysis is crucial for maintaining quality of service (QoS) in heterogeneous networks and optimizing signal processing for multiuser MIMO uplink channels [16, 17]. It also contributes to spectral efficiency in multicell networks with imperfect channel state information (CSI) [18] and is essential for the joint design of MIMO and visible light communication (VLC) systems [19].

The necessity for robust wireless channel values is evident as it significantly impacts the efficacy of signal processing algorithms, which are assessed by the MSE metric. This research provides a comprehensive MSE analysis, along with different MIMO antenna configurations in Rayleigh and Rician fading channels.

The structure of this research article is as follows: Section 1 introduces the fundamentals of signal processing for MIMO wireless transceivers. Section 2 mathematically formulates the system model for MIMO wireless transceivers. Section 3 discusses the signal processing algorithms employed in MIMO wireless transceivers, including LS, MMSE, and ML approaches. Section 4 presents the simulation results for the MSE metric in MIMO wireless transceivers. Section 5 concludes the paper.

In this research article, matrices are denoted by boldface capital letters, vectors are indicated by boldface lowercase letters, and scalars are presented in standard font. Hermitian, transpose, complex conjugate, expectation, pseudo-inverse, inverse, trace, and the Frobenius norm of a matrix 'X' are represented as: x^H, x^T, x^*, E{.}, x⁺, x^-1, trace{}, $\| \,\|_F^2$ in this research paper.

Mean square error can be formulated using signal processing algorithms such as least squares by considering the MIMO received signal matrix as stated earlier. To determine the mean square error for least squares (LS) signal processing algorithm the objective function is:

$K=(\boldsymbol{R}-\boldsymbol{C D})^2$          (2)

Executing partial differentiation on considering the channel parameter C it can be given as:

$\frac{\partial K}{\partial \boldsymbol{C}}=2(\boldsymbol{R}-\boldsymbol{C D})(-\boldsymbol{D})$       (3)

To obtain its corresponding estimate of the estimation parameter C it is proceeded as:

$0=2(\boldsymbol{R}-\widehat{\boldsymbol{C}} \boldsymbol{D})(-\boldsymbol{D})$          (4)

On simplifying further the estimated value is:

$\widehat{C_{L S}}=\boldsymbol{R} \boldsymbol{D}^{-1}$             (5)

For the above given estimate the mean square error (MSE) pertaining to least squares signal processing algorithm are determined as per the representation:

$M S E_{L S}=\left\{\operatorname{tr}\left\{E\left\{\left(\boldsymbol{E} \boldsymbol{E}^H\right)\right\}\right\}\right\}$          (6) 

where, E is the error matrix due to the least squares estimator, which is:

$\boldsymbol{E}=\boldsymbol{C}-\widehat{\boldsymbol{C}}_{L S}$          (7)

Substituting the value of error matrix in above MSE equation it is:

$M S E_{L S}=\left\{\operatorname{tr}\left\{E\left\{\left(\left(\boldsymbol{C}-\widehat{\boldsymbol{C}}_{L S}\right)\left(\boldsymbol{C}-\widehat{\boldsymbol{C}}_{L S}\right)^H\right)\right\}\right\}\right\}$         (8)

Using the estimated value of LS signal processing algorithm in the above equation it is postulated as:

$M S E_{L S}=\left\{\operatorname{tr}\left\{E\left\{\left(\left(\boldsymbol{C}-\boldsymbol{R} \boldsymbol{D}^{-1}\right)\left(\boldsymbol{C}-\boldsymbol{R} \boldsymbol{D}^{-1}\right)^H\right)\right\}\right\}\right\}$          (9)

In order to derive the MSE for minimum mean square error (MMSE) signal processing algorithm [8] it is C_MMSE=RV₀. where $\boldsymbol{V}_0=\operatorname{argmin} E\left\{\|\boldsymbol{C}-\widehat{\boldsymbol{C}}\|^2\right\}$ and it can also be written as $\boldsymbol{V}_0=\operatorname{argmin} E\left\{\|\boldsymbol{C}-\boldsymbol{R} \boldsymbol{V}\|_F^2\right\}$ with respect to a matrix V. The estimation error is given as:

$\varepsilon=E\left\{\|\boldsymbol{C}-\boldsymbol{R} \boldsymbol{V}\|_F^2\right\}$          (10)

Expanding further it is given as:

$\begin{aligned} & \varepsilon=\operatorname{tr}\left\{\boldsymbol{R}_C\right\}-\operatorname{tr}\left\{\boldsymbol{R}_C \boldsymbol{D} \boldsymbol{V}\right\}-\operatorname{tr}\left\{\boldsymbol{C}^H \boldsymbol{D}^H \boldsymbol{R}_C\right\} \\ &+ \operatorname{tr}\left\{\boldsymbol{V}^{\boldsymbol{H}}\left(\boldsymbol{D}^H \boldsymbol{C}^H \boldsymbol{D}+\sigma_n^2 T_r \boldsymbol{I}\right) \boldsymbol{V}\right\} +\operatorname{tr}\left\{V^H\left(D^H C^H D+\sigma_n^2 T_r I\right) V\right\}\end{aligned}$        (11)

where,

$\boldsymbol{V}_0=\left(\boldsymbol{D}^H \boldsymbol{R}_C \boldsymbol{D}+\sigma_n^2 \boldsymbol{T}_r \boldsymbol{I}\right)^{-1} \boldsymbol{D}^H \boldsymbol{R}_C$         (12)

The MMSE signal processing algorithm is given as:

$\boldsymbol{C}_{M M S E}^{\wedge}=\boldsymbol{R}\left(\left(\boldsymbol{D}^H \boldsymbol{R}_C \boldsymbol{D}+\sigma_n^2 T_r \boldsymbol{I}\right)^{-1} \boldsymbol{D}^H \boldsymbol{R}_C\right)$        (13)

Based on the above representation estimate the mean square error (MSE) pertaining to MMSE signal processing algorithm is determined as per the representation:

$M S E_{M M S E}=\left\{\operatorname{tr}\left\{E\left\{\left(\boldsymbol{E E}^H\right)\right\}\right\}\right\}$       (14)

where, E is the error matrix due to the MMSE signal processing algorithm, which is given as:

$\boldsymbol{E}=\boldsymbol{C}-\widehat{\boldsymbol{C}}_{M M S E}$        (15)

Substituting the value of error matrix in above MSE equation it reaches to:

$M S E_{M M S E}=\left\{\operatorname{tr}\left\{E\left\{\left(\left(\boldsymbol{C}-\widehat{\boldsymbol{C}}_{M M S E}\right)\left(\boldsymbol{C}-\widehat{\boldsymbol{C}}_{M M S E}\right)^H\right)\right\}\right\}\right\}$        (16)

Now to derive maximum likelihood (ML) signal processing algorithm [9] for MIMO wireless transceiver for analysis of mean square error from equation (1) the probability density function (PDF) is defined as:

$p(\boldsymbol{R} ; \boldsymbol{C})=\frac{1}{(2 \pi)^{\frac{N}{2}} \operatorname{det}(\boldsymbol{C})} exp \left\{-\frac{1}{2}(\boldsymbol{R}-\boldsymbol{C D})^T \boldsymbol{R}_c^{-1}(\boldsymbol{R}-\boldsymbol{C D})\right\}$          (17)

Proceeding to find the maximum likelihood estimate consider the objective function:

$K(\boldsymbol{C})=\left\{(\boldsymbol{R}-\boldsymbol{C} \boldsymbol{D})^T \boldsymbol{R}_c^{-1}(\boldsymbol{R}-\boldsymbol{C} \boldsymbol{D})\right\}$         (18)

The above represents a quadratic function of elements of C and $R_c^{-1}$ is a positive definite matrix on C. Taking partial differentiation with respect to C it is given as:

$\frac{\partial l n}{\partial \boldsymbol{C}}=\frac{\partial}{\partial \boldsymbol{C}}\left\{(\boldsymbol{R}-\boldsymbol{C} \boldsymbol{D})^T \boldsymbol{R}_c^{-1}(\boldsymbol{R}-\boldsymbol{C} \boldsymbol{D})\right\}$          (19)

On setting the gradient equal to zero, it reaches to:

$\boldsymbol{D}^T \boldsymbol{R}_c{ }^{-1}\left(\boldsymbol{R}-\widehat{\boldsymbol{C}}_{M L} \boldsymbol{D}\right)=0$          (20)

Multiplying further the above equation is:

$\boldsymbol{D}^T \boldsymbol{R}_c{ }^{-1} \boldsymbol{R}-\boldsymbol{D}^T \boldsymbol{R}_c{ }^{-1} \widehat{\boldsymbol{C}}_{M L} \boldsymbol{D}=0$         (21)

Rearranging to find the maximum likelihood estimation signal processing algorithm it is further given as:

$\boldsymbol{D}^T \boldsymbol{R}_C{ }^{-1} \boldsymbol{R}=\boldsymbol{D}^T \boldsymbol{R}_C{ }^{-1} \widehat{\boldsymbol{C}}_{M L} \boldsymbol{D}$         (22)

There ML estimation signal processing algorithm is:

$\frac{\boldsymbol{D}^T \boldsymbol{R}_C{ }^{-1} \boldsymbol{R}}{\boldsymbol{D}^T \boldsymbol{R}_C{ }^{-1} \boldsymbol{D}}=\hat{C}_{M L}$          (23)

Taking the denominator term to the numerator it is:

$\left(\boldsymbol{D}^T \boldsymbol{R}_{\boldsymbol{c}}{ }^{-1} \boldsymbol{D}\right)^{-1} \boldsymbol{D}^T R_C{ }^{-1} \boldsymbol{R}=\widehat{\boldsymbol{C}}_{M L}$           (24)

Finally, the maximum likelihood estimation algorithm is given as:

$\widehat{\boldsymbol{C}}_{M L}=\left(\boldsymbol{D}^T \boldsymbol{R}_C{ }^{-1} \boldsymbol{D}\right)^{-1} \boldsymbol{D}^T \boldsymbol{R}_C{ }^{-1} \boldsymbol{R}$         (25)

Similar to representation of the estimate of LS and MMSE signal processing algorithm the mean square error (MSE) pertaining to ML signal processing algorithm is determined as per the representation:

$M S E_{M L}=\left\{\operatorname{tr}\left\{E\left\{\left(\boldsymbol{E E}^H\right)\right\}\right\}\right\}$        (26)

where, E is the error matrix due to the ML signal processing algorithm, which is given as:

$\boldsymbol{E}=\boldsymbol{C}-\widehat{\boldsymbol{C}}_{M L}$            (27)

Substituting the value of error matrix in above MSE equation it is found to be as:

$M S E_{M L}=\left\{\operatorname{tr}\left\{E\left\{\left(\left(\boldsymbol{C}-\widehat{\boldsymbol{C}}_{M L}\right)\left(\boldsymbol{C}-\widehat{\boldsymbol{C}}_{M L}\right)^H\right)\right\}\right\}\right\}$          (28)