D2L2-Dense LSTM Deep Learning Based Nonlinear Acoustic Echo Cancellation

3.1 System model

The nonlinear AEC's system model is shown in Figure 1. The room impulse response (RIR) of h(n) is combined with the far-end signal, x(n), to produce an echo signal that is mixed with the near-end signal of s(n). The three components of echo, near-end, and noise signals construct the microphone signal y(n), as shown by:

$y(n)=s(n)+d(n)+v(n)$     (1)

1.png

Figure 1. Nonlinear acoustic echo cancellation system

The AEC’s goal is to reduce echo d(n) and transfer near-end signal s(n) towards the far-end. Adaptive filters are utilized in classical AEC methods to estimate the IR $\hat{h}(n)$, where y(n) represents the microphone signal and x(n) represents the far-end signal. The estimated signal $\hat{d}(n)$ of the echo is then calculated as follows:

$\hat{d}(n)=(\hat{h}(n))^T \mathrm{x}(n)$     (2)

where, x(n) is the input buffered, $\hat{h}(n)$ is the adaptive filter of length L, and T denotes transpose. The microphone signal is subtracted from $\hat{d}(n)$ to yield the system output. This output signal or the error signal e(n) is defined as:

$e(n)=y(n)-\hat{d}(n)$     (3)

The estimation of the acoustic route h(n) is the crucial step in AEC methods. Traditional AEC techniques cannot manage nonlinear aberrations because they are inherently linear systems. Fast transversal filters [19], adaptive Volterra filters [20], Subband filters [21], and Functional link AF [22] explicitly represent the AEC system’s nonlinearity. However, in actual situations with large nonlinearities and potent interference signals, their performance is constrained.

The AEC’s ultimate objective is to eliminate both the end signal of the far-end and the circumstantial noise at the end of the near-end, allowing only the signal at the near-end to be sent to the far-end. As per speech separation [23], AEC can be conceptualized as a separation issue, with the primary objective being the isolation of the near-end speech, which constitutes the main source of interference in the microphone recording. To address this, we utilize controlled speech separation to process the microphone inputs and simultaneously extract both the original near-end signal and far-end signal, instead of directly predicting the acoustic echo path. The clear near-end signal from the microphone signal is extracted using a dense LSTM architecture that is based on deep learning and is covered in the next section.

3.2 D²L² dense LSTM deep learning-based NAEC

Here we incorporate a deep learning approach into NAEC for audio echo cancellation. Figure 2 is a diagram that illustrates our suggested method. The schematics use a fully connected input layer with 1024 units that passes the signal to three BILSTM layers with 512 units in each layer and a fully connected layer with 600 units. The signal is then passed on to an output layer with 512 units.

2.png

Figure 2. Proposed D²L²: Dense LSTM deep learning based NAEC

Broadly, the approach aims to mitigate background noise and acoustic echo, facilitating the extraction of near-end speech. This involves estimating both the magnitude mask (MM) for the microphone spectral signal and the binary mask for near-end speech (NSM) from x(n) and y(n). More precisely, a dense LSTM receives acoustic features derived initially from both the microphone signal and the far-end signal. To estimate the MM and NSM, the top layer of the LSTM functions as a mask estimation layer. The near-end signal estimate is then resynthesized using the mask estimate to the microphone signal.

3.3 STFT-based spectral feature extraction

Generally, speech signals are not stationary. Often, we want to examine the spectrum of each phoneme independently, but if we convert a spoken phrase to the frequency domain, we acquire a spectrum that is an average of all phonemes in the sentence. We can concentrate on the signal's characteristics at a certain moment by segmenting the signal into shorter chunks. We acquire the STFT of the signal by windowing and taking the Discrete Fourier transform (DFT) of each window.

By selecting an appropriate window function, a time-domain STFT partitions an acoustic wave into segments, revealing how frequency components evolve. An advantage of STFTs is that their parameters possess a physical and intuitive interpretation, corresponding directly to the spectrum.

Specifically, for an input signal s(n) and window w(n), the transform is defined as,

$S(m, k)=\sum_n s(m, n) e^{-\frac{j 2 \pi n k}{N}}$     (4)

$s(m, n)=s(n) * w(n-m L)$     (5)

where, the fast Fourier transform (FFT) size is represented in terms of 2 orders and denoted here as N, n is the sample time index within the given frame, m is the frame index and w(n) is the windowing callback with N signals. w(n) is positioned at shifting step size L, and $1-\frac{L}{N}$ is the overlay ratio among continuous frames. The input signals are presented into 16 ms frames with an 8 ms delay between successive frames after being sampled at 16kHz. Every frame is then subjected to a 512-point STFT to obtain a vector of spectral magnitudes, which results in 256 frequencies. To form the input feature vector, we concatenate the STFT features extracted from the far-end signal $F_x(t)$ and the microphone signal $F_y(t)$ as follows:

$F(t)=\left[F_y(t), F_x(t)\right]^T$     (6)

where, the feature vectors of the microphone signal and far-end speech at time frame m are indicated by the symbols $F_y(t)$, $F_x(t)$, as a result, 1024 (512×2) is the input's dimensionality. The STFT spectrogram is shown in Figure 3.

3.png

Figure 3. STFT spectrogram of the signal

3.4 D²L² network design

RNN is a type of model for recording chronological data and is used to tackle a variety of issues, including speech recognition [24], machine translation [25], and natural language processing [26]. The sequential data have long-term dependence, which leads to disappearing and expanding gradient difficulties. This problem can be resolved by the application of LSTM and gated RNNs. In the LSTM network, supplementary memory cells and gate procedures exist. At the time of backpropagation, moving out of the gradient will be precluded by the gate operations. With fewer gate functions, the gated recurrent unit (GRU) performs similarly [27].

Gated commentary: Two successive time steps are fully connected by an RNN. The depths of the gated feedback (FB), feedforward (FF), and recurrent feedback may all be modeled together since their connections to the model with orthogonal depths do not overlap. We suggest an attention gate with these three features, which regulates the flows from each state to improve the overall performance via D²L². Figure 4 depicts deep learning methodology and structure.

4.png

Figure 4. Proposed LSTM

3.5 LSTM-based deep learning

In the general structure of the recurrent layer, the preceding hidden state $h_{t-1}$ and input $x_t$ are used to define each hidden state at time t as $h_t$, which is given by:

$h_t=\phi\left(h_{t-1}, x_t\right)=\phi\left(W x_t+U h_{t-1}\right)$     (7)    

where, $\phi$ is the "tanh" nonlinear process with each sample operation, W is the recurrent layer weight matrix and U is the feed-forward weight matrix. For a typical RNN architecture, the last hidden state $h_{t-1}$ will store the previous state inputs for further calculations. This leads to poor memory processing when the hidden state is allocated less storage space, which is reflected in the continuity of the state variables. To overcome this problem, a stacked RNN is employed to capture prolonged dependencies across multiple state statuses within the hidden layers, as illustrated below:

$h_t^j=\phi\left(W^j h_t^{j-1}+U^{j \rightarrow j} h_{t-1}^j\right)$     (8)

where, $U^j$ is the changeover from time step t-1 to time step t at layer j, which impacts the weight matrix, and $W^j$ is the transition from layer j-1 to layer j, which affects the weight matrix. The sequential data may be modeled across various timeframes using the stacked RNN. When prediction data are stacked at the top layer, long-term (LT) dependencies are highly covered by hidden state storage. Hence, deep recurrent network formation is necessary by connecting the current hidden state to the previous ones. Thus, we can improve the long-term dependency [28], and it is represented as:

$h_t^j=\phi\left(W^j h_t^{j-1}+\sum_{k=1}^K U^{(k, j) \rightarrow j} h_{t-k}^j\right)$     (9)

where, K is the recurrent depth and $U^{(k, j) \rightarrow j}$ is the weight matrix of layer j with the transition occurring from time t-k to time t. The shortcut routes from many earlier concealed states are made by direct linkages. The model with shortcut pathways allows access to earlier hidden states farther away from $h_t^j$ with the same number of transitions as the model without shortcut paths. Many recurrent models typically establish connections solely among hidden states within the same layer. However, the network layer adaptly mitigates numerous timing issues by aggregating feedback connections from preceding hidden states $h_{t-1}^j$ to the hidden state $h_t^j$ across different layers of $h_t^j$, as demonstrated below:

$h_t^j=\phi\left(W^j h_t^{j-1}+\sum_{i=1}^L U^{i \rightarrow j} h_{t-1}^i\right)$     (10)

where, the feedforward depth is L and $U^{i \rightarrow j}$ is the weight matrix transition from time t-1 to t. The global gate is designed to regulate the number of streams between diverse hidden states with diverse time measures [27].

$h_t^j=\phi\left(W^j h_t^{j-1}+\sum_{i=1}^L g^{i \rightarrow j} U^{i \rightarrow j} h_{t-1}^i\right)$     (11)

The gate controlling the flow of information from each hidden state (HS) to multiple preceding HSs across all layers is denoted as $g^{i \rightarrow j}$.

$g^{i \rightarrow j}=\sigma\left(w_g h_t^{j-1}+u_g^{i \rightarrow j} h_{t-1}^*\right)$     (12)

where, $h_t^{j-1}$ is the same size as the state vector $w_g$ and $h_{t-1}^*$ is the size of the state vector $u_g^{i \rightarrow j}$. It has been established that for a concatenated vector comprising all hidden states from the mentioned time step, the sigmoid function applied elementwise to the vector is denoted by *.

In gated LSTM, the output gate $o_t^j$, forget gate $f_t^j$, input gate $i_t^j$, and memory cell gate $\tilde{c}_t^j$ are defined as follows:

$i_t^j=\sigma\left(W_i^j h_t^{j-1}+U_i^{j \rightarrow j} h_{t-1}^j\right)$     (13)

$f_t^j=\sigma\left(W_f^j h_t^{j-1}+U_f^{j \rightarrow j} h_{t-1}^j\right)$     (14)

$o_t^j=\sigma\left(W_o^j h_t^{j-1}+U_o^{j \rightarrow j} h_{t-1}^j\right)$     (15)

$\tilde{c}_t^j=\phi\left(W_c^j h_t^{j-1}+\sum_{i=1}^L g^{i \rightarrow j} U_c^{i \rightarrow j} h_{t-1}^j\right)$     (16)

Like in conservative LSTM, the memory cell gate $\tilde{c}_t^j$ has a feedback loop for gates in gated LSTM. Using gates and the memory cell state in (13)-(16), the new hidden state $h_t^j$ and new memory cell state $c_t^j$ are modeled as

$c_t^j=f_t^j \cdot c_{t-1}^j+i_t^j \cdot \tilde{c}_t^j$     (17)

$h_t^j=o_t^j \cdot \phi\left(c_t^j\right)$     (18)

where, the default dot product denotes matrix multiplication.

3.6 Modelling of D²L²

Deep learning networks can be trained more effectively when skip-linking state connections are incorporated as they enable connections to bypass certain layers. This approach which involves Skip linking state connections to feedback connections has been utilized in various studies [29, 30]. We apply skip-linking state connections to recurrent connections in this work. The skip-linking state connections over time are equivalent to the shortcuts from the various HSs that came before to the hidden state at time t. The feedback linkages between several layers of various time steps are among the shortcut routes. The attention gate normalizes each connection, resembling closely the global gated feedback RNN. The dense recurrent neural network is represented as follows:

$h_t^j=\phi\left(W^j h_t^{j-1}+\sum_{k=1}^K \sum_{i=1}^L g^{(k, i) \rightarrow j} U^{(k, i) \rightarrow j} h_{t-k}^i\right)$     (19)

where, the attention gate is mathematically denoted as $g^{(k, i) \rightarrow j}$ below:

$g^{(k, i) \rightarrow j}=\sigma\left(w_g^i h_t^{j-1}+u_g^{(k, i) \rightarrow j} h_{t-k}^i\right)$     (20)

where, (20) is a gate callback of the foregoing HS at time t-k and layer i, and (12) is a callback of all the interconnected foregoing HSs. The dense recurrent neural network is straightforwardly extended to dense LSTM and is described as follows:

$i_t^j=\sigma\left(W_i^j h_t^{j-1}+\sum_{k=1}^K \sum_{i=1}^L g_i^{(k, i) \rightarrow j} U_i^{(k, i) \rightarrow j} h_{t-k}^i\right)$     (21)

$f_t^j=\sigma\left(W_f^j h_t^{j-1}+\sum_{k=1}^K \sum_{i=1}^L g_f^{(k, i) \rightarrow j} U_f^{(k, i) \rightarrow j} h_{t-k}^i\right)$     (22)

$o_t^j=\sigma\left(W_o^j h_t^{j-1}+\sum_{k=1}^K \sum_{i=1}^L g_o^{(k, i) \rightarrow j} U_o^{(k, i) \rightarrow j} h_{t-1}^i\right)$     (23)

$\tilde{c}_t^j=\phi\left(W_c^j h_t^{j-1}+\sum_{k=1}^K \sum_{i=1}^L g_c^{(k, i) \rightarrow j} U_c^{(k, i) \rightarrow j} h_{t-1}^i\right)$     (24)

In contrast to the gated FB-LSTM, the parameter g is distributed across all memory cell states and gates. We examined the spontaneous adjustment of dense connections. Figure 5 illustrates the state connection diagram of dense LSTM. Figure 5 (a) shows the conventional LSTM unfolded in time, and Figure 5 (b) represents the gated feedback LSTM. Figure 5 (c) shows the preceding state connections across the LSTM, and Figure 5 (d) shows the dense LSTM related with Figure 5 (b) and Figure 5 (c). HSs are marked in red. The links of dense LSTM utilized in FF steps are highlighted in bold. The FB-states between the higher and inferior layers are represented in yellow.

51.png

52.png

53.png

54.png

55.png

Figure 5. Dense LSTM state representations

3.7 D²L² training targets

For training the proposed novel D²L² algorithm, training inputs and training targets are constructed. The extracted STFT magnitude features are the input for training in D²L². There are mainly two training targets. One of the targets is the MM of spectral extracted far-end and microphone signals. The second target is the NSM.

In this section, we experiment with our proposed D²L² algorithm for NAEC and compare it with previous adaptive filter-based echo cancellation methods from recent works, such as NSAEC, KIPNLMS, and PFLAF [33-35]. The key metrics for AEC's performance are the convergence time and the echo return loss enhancement (ERLE).

ERLE variation is frequently used to assess the performance of AEC systems when there is background noise, and it is defined as:

$E R L E=10 \log _{10}\left[\frac{\sum_n y^2(n)}{\sum_n \hat{S}^2(n)}\right]$     (29)

It measures the reduction in echo after applying the cancellation algorithm.

The cross-validation technique is used to assess the stability and generalization performance of the algorithm. This is done by dividing the data set into training and validation sets multiple times. The training set is used to train the model, while the validation set is used to evaluate its performance.

6.png

Figure 6. Estimation of ERLE for an echo canceled signal

Figure 7. Non-linear echo cancellation results of proposed D²L²