Fusion of Dual Sensor Features for Fall Risk Assessment with Improved Attention Mechanism

Falls are recognized as the leading cause of accidental deaths and injuries among people over the age of 65 years. Reduced mobility and impaired balance make older adults more prone to falls, which threatens their safety. Therefore, it is important to conduct a risk assessment for falls to predict the risk of falls in time and to take active preventive and intervention measures to minimize falls in the elderly. The purpose of fall risk assessment is to identify the elderly with high risk of falls and to propose individualized fall prevention methods, so as to we hope to reduce or avoid the occurrence of fall accidents. Clinical assessment of fall risk mainly uses fall risk scales such as the Berg Balance Scale, Tinetti Balance and Gait Scale, and the TUGT test [1, 2]. These methods require doctors to score, so it takes a long time and the process is complicated. With the development of technologies such as the Internet of Things and artificial intelligence, researchers have started to monitor human gait with wearable devices, by analyzing kinematic and kinetic characteristics such as acceleration, rhythm, gait speed, plantar pressure, and step length, to assess fall risks [3]. Gait is the posture and state of the human body when walking. Gait analysis can be used to effectively evaluate the risk of falls. The selection of gait features directly affected the assessment accuracy. Identifying the gait features with a high risk of falls can help doctors objectively evaluate the walking ability of the elderly in clinics. The preventive program has important practical significance for scientifically guiding the elderly to walk with a minimized fall risk.

Among the many wearable sensors, IMU and plantar pressure sensors are widely used in fall risk assessment because of their small size, low price and little impact on wearers and that IMU is ideal devices for detecting gait in daily life. However, since the IMU signals only capture gait kinematic parameters, the model's performance for fall risk assessment is poor [4-7]. Although the plantar pressure signal has rich dynamic gait information, research on fall risk assessment based on plantar pressure is still in its early stages. If only a single IMU or plantar pressure signal is used, there is a problem of insufficient robustness. Furthermore, most of the construction of risk assessment models lacks real-life data in daily scenarios, resulting in a lack of effectiveness in the risk assessment model [8, 9].

Therefore, the existing risk assessment model lacks validity, and the single-mode IMU and plantar pressure signals used alone have poor robustness. We proposed a new method based on the ISA-ECA-CNN-BiLSTM model with an improved attention mechanism, to perform fall risk assessment through adaptive weighted feature fusion based on the collected IMU signal and plantar pressure signal datasets.

1.1 Related work

Scholars have proposed a various fall risk assessment methods that use wearable devices. Among these methods and their variants, methods based on inertial sensors and plantar pressure sensors are the most attractive, as inertial sensors and plantar pressure transmitters can be mounted on the body and collect very detailed movement characteristics [10-16], and the acquired time-series gait data can be used to identify and authenticate people [17, 18]. Initially, machine learning methods were adopted for fall risk assessment. Wang et al. [19] designed an insole pressure sensor system that could provide users with real-time predictions and early warnings. It can predict the risk of falls in advance, allowing users sufficient time to take effective intervention measures and effectively reduce the occurrence of falls. Yang et al. [20] collected acceleration, angular velocity, and plantar pressure data of nine common lower limb movements of 10 subjects, extracted 13 gait features as the input of a common classifier to identify motion postures, and found that KNN has the highest classification accuracy. The recognition rate of 9 kinds of common types was high as 99.96%. Yu et al. [21] designed a plantar pressure sensor-based inversion-detection system. The fuzzy inference algorithm is used to divide the gait phase and then detect whether the inversion of the foot occurs, and the experimental results show that the plantar pressure data have an important predictive value for sports risk. Zhao et al. [22] developed a wearable plantar pressure force measurement and analysis (WPPFMA) system based on a flexible sensor matrix film to monitor plantar pressure force in real time. This system is highly valuable for personal foot care, gait analysis, and clinical diagnosis. Peng et al. [23] used an integrated empirical modal decomposition to decompose Parkinsonist left and right VGRF signals from high to low frequency bands into intrinsic modal functions. A person's gait health is then analyzed using their instantaneous phase dependence to assess the strength of the causal interaction between each intrinsic mode function. Gao et al. [24] collected the foot low-pressure data of 48 subjects, extracted 44 multi-dimensional weak foot features based on one legged COP, and analyzed the symmetry and time consistency of the gait line by using the probability distribution method. The final model achieved an accuracy of 87.5% for the test data. These methods can differentiate individual features with transparency, and the individual contribution of features to the decision is visible. However, manual feature extraction relies heavily on human experience, which is time-consuming and inefficient in fall prediction [25, 26]. Deep learning models can gradually extract higher-level features from raw inputs; therefore, there is no need to manually select relevant features that may require expert knowledge. In particular, for gait analysis, deep learning can handle complex data and provide accurate results. For example, Nait Aicha et al. [27] first studied CNN and LSTM networks that can automatically extract features from raw accelerometer data for fall risk assessment. Experiments showed that this method has a higher prediction accuracy than CNN and LSTM alone. Tunca et al. [28] used an inertial sensor-based gait analysis system with a LSTM neural network to extract gait spatiotemporal parameter sequences as input features for fall risk assessment and achieved an excellent classification accuracy of 92.1%. Baloch et al. [29] performed early fusion and late fusion based on acceleration sensors and angular velocity sensors to identify human activity using CNN and convolutional CNN-LSTM. A weighted f1 score of 93.78% was obtained on the RealDisp dataset when late fusion was performed. Liang et al. [30] proposed a fall risk prediction model for the elderly using the plantar center of force with the ConvLSTM algorithm. This method achieved an accuracy of 94 percent accuracy. Yu et al. [31] used three deep networks, CNN, LSTM, and Hybrid Convolutional Long Short-Term Memory (ConvLSTM), to construct a fall prediction model, with model inputs of raw IMU data. The results on the SisFall dataset showed that the hybrid ConvLSTM model had an average of non-fall, pre-impact fall, and fall sensitivity of 93.15%, 93.78%, and 96.00%, respectively, which are higher than the LSTM (except for the, fall class) and CNN models. Shalin et al. [3] used 16 gait features extracted from plantar pressure data for gait freezing detection, the study collected plantar pressure data from 11 Parkinson's disease and classified them using the LSTM network. The experimental results showed that the classification model achieved 82.1% (SD 6.2%) mean sensitivity and 89.5% (SD 3.6%) mean specificity.

1.2 Motivation and contribution

In summary, currently, IMU-based fall risk assessment methods still have the problem of low accuracy, and there are relatively few fall risk assessment methods based on plantar pressure sensors. It is still debatable how to improve the accuracy and reliability of fall risk assessment. In addition, IMU can acquire gait kinematic parameters, and plantar pressure sensor contains rich gait dynamics information, combining IMU and plantar pressure sensor may be an effective way to improve fall risk assessment. So Based on the above, this study establishes a dataset containing multi-sensor information for fall risk assessment and proposes a fall risk assessment model that fuses multi-modal data and achieves 98.4% accuracy.

The main innovations and contributions of this work include:

We built a dataset that includes labels indicating fall risk.

We proposed a deep learning framework incorporating an improved attention mechanism to better focus on global features, thereby improving recognition accuracy.

By employing adaptive weighting, we fused signals from two sensors at the feature level, ensuring a higher recognition accuracy and enhancing the robust-ness of the fall risk assessment model.

3.1 Improved ECA attention mechanism

Wang et al. [41] believed that the compression and dimensionality reduction of SE (Squeeze and Excitation) attention model has a negative impact on learning the dependencies between channels, and capturing the dependencies between all channels is inefficient and unnecessary. Therefore, an ECA attention mechanism was proposed, as shown in Figure 5. The ECA module proposes a local cross-channel interaction strategy without a dimensionality reduction. The ECA attention mechanism uses 1-dimensional convolution to effectively realize partial cross-channel interaction and obtain dependencies between channels, thereby avoiding the negative effects of dimensionality reduction. The ECA mechanism plays a crucial role in enhancing the accuracy of fall risk assessment by focusing on the most informative channels within the input data. This selective attention mechanism helps highlight the key features that distinguish an individual's gait. Additionally, it aids in reducing the impact of irrelevant or noisy data, ultimately improving the accuracy and robustness of the fall risk assessment model. However, the ECA attention mechanism ignores the spatial attention module.

5.png

Figure 5. Schematic diagram of the ECA attention mechanism module

6.png

Figure 6. Schematic diagram of the improved ECA attention mechanism module (ISA-ECA)

Therefore, we have introduced an enhanced ECA attention mechanism, as delineated in Figure 6. Following the ECA operation, a spatial attention module has been incorporated. The output feature map, enriched with channel attention, is fed into this spatial attention module. Within the spatial attention module, initial operations entail average pooling and maximum pooling along the channel axis of the input feature map, yielding intermediate results denoted as A and B, respectively. Subsequently, these outcomes undergo connections and convolutions via a conventional convolutional layer, thereby generating a 2D spatial attention map. The mathematical formulation of the spatial attention module is expressed as follows. This innovative augmentation aims to address issues of redundancy and elevate the discriminative power of the attention mechanism for improved performance in the given context.

$\begin{aligned} M_S(F)=\sigma\left(f^{7 \times 7}\right. & ([\operatorname{AvgPool}(F) ; \operatorname{MaxPool}(F)])) =\sigma\left(f^{7 \times 7}\left(\left[F_{\text {avg }}^S ; F_{\max }^S\right]\right)\right)\end{aligned}$           (1)

Among them, $f^{7 \times 7}$ represents the convolution operation with a convolution kernel size of 7*7, and $\sigma$ represents the Sigmoid function. $\mathrm{F}_{\mathrm{avg}}^{\mathrm{S}}$ and $\mathrm{F}_{\mathrm{max}}^{\mathrm{S}}$ represent average pooling features and maximum pooling features, respectively.

The integration of the spatial attention module serves as a remedial measure for the limitations inherent in the ECA attention mechanism. This enhancement facilitates a more precise extraction of human gait features, thereby contributing to an elevated model accuracy. The incorporation of the spatial attention module is a strategic refinement aimed at mitigating deficiencies in the original ECA mechanism, ultimately bolstering the discernment and fidelity of the model's representations of intricate gait patterns.

3.2 ISA-ECA-CNN-BiLSTM model architecture

The overall process of fall gait risk assessment proposed in this paper is shown in Figure 7. Firstly, data collection is carried out through human walking experiments, fixing the IMU and plantar pressure sensor on human’s sole and the calf respectively, the gait sequence data set was constructed by filtering and segmenting the IMU and plantar pressure signals. Then the gait data set was inputted into the ISA-ECA-CNN-BiLSTM network model for the gait detection task. The main network structure of the model is a CNN module and a Bi-LSTM module, where the CNN module is used to provide an abstract representation of the gait signal in the feature map. LSTM is used to model the feature maps output by the CNN module. The translation invariance of the convolution operation in the convolutional layer ensures that the key timing of the features is preserved while extracting low-level local features, that is why the output vector of the CNN module can be used as the input of the LSTM module. After the multi-layer convolution operation, the field of view of the convolution kernel is expanded, and the length of the feature sequence is shortened, which realizes the compression of the original information and is more conducive to the modeling of the LSTM network. In addition, max pooling and improved ECA attention are added after the second or fourth layers of convolutional blocks. This operation can fuse channel and spatial global information during local feature extraction, improving the ability to represent gait signals. Finally, the features extracted from the IMU signal and the plantar pressure signal are fused with adaptive weighted features to perform the fall risk detection task. The framework of the fall gait risk analysis model is shown in Figure 8.

The ISA-ECA-CNN-BiLSTM model consists of the following seven parts:

Input layer: Acceleration and angular velocity data and plantar pressure data, pre-processed and fed into the target network.

Convolution layer: In time series data, adjacent signals can be correlated with each other, and CNN can capture the local correlation of time series data; therefore, we used it to extract local features. Each convolution layer contains 64 filters and the kernel size is 7×1. Each unit is activated according to the ReLU activation function after convolution.

Pooling layer: The second and fourth layer convolutions of the model used the maximum pooling method for sub-sampling, further reducing the dimensionality of the information extracted by the convolutional layer, the size of the model and the number of calculations to prevent overfitting. The size of the kernel in this layer is 2×2, the stride is 2.

Attention Mechanism: The attention mechanism can make the model more accurate and efficient in processing sequence data. In the attention mechanism, the output of each neuron depends not only on the output of all the neurons in the previous layer, but can also be weighted according to different parts of the input data. In order to better extract features of human gait, we added the above-mentioned improved ECA module at different positions of the model.

Bidirectional LSTM layer: The input sequences are input into two LSTM neural networks in positive and reverse order respectively for feature extraction. The word vector is formed after the two output vectors are spliced together as the final feature expression of the word. The final prediction result is determined by both forward propagation and back propagation.

7.png

Figure 7. Fall risk assessment framework

8.png

Figure 8. Schematic diagram of model frame of ISA-ECA-CNN-BiLSTM

Dropout layer: It is added after the fully connected layer and bidirectional LSTM layers. When propagating forward, it makes the activation value of a neuron stop working with a certain probability p, which makes the model more generalized, we set the value of dropout to 0.5.

Output layer: The features processed by the designed network are connected to each node and all nodes of the previous layer through the fully connected layer to synthesize the features extracted before, and after obtaining a specific value, the final classification result is obtained based on the softmax classifier.

In this paper, we used the above model for fall risk assessment using a multimodal fusion approach. The conventional approach for combining multimodal features is to concatenate the feature vectors of each modality directly. Employing a direct concatenation of feature vectors may not be the most optimal approach due to the inherent dissimilarity in the contributions of plantar pressure signals and IMU signals to the fall risk assessment task. Additionally, there exists a likelihood of high redundancy or, conversely, complementary relationships between features extracted from these two signal sources. This elevated duplication rate underscores the necessity for a more nuanced strategy that captures the nuanced interplay and unique characteristics of each signal type in order to enhance the overall effectiveness of fall risk assessment. Therefore, this paper adopts adaptive feature weighted fusion, and automatically shields redundant features and highlights complementary features through network training to achieve feature fusion of the two signals. We assigned a weight to each feature parameter to indicate how important that feature parameter is for the fall risk assessment task. The weight value can be learned automatically through neural network training, and automatically assign higher weights to the feature parameters with high contribution degree.

The computational process of adaptive weighted feature fusion is shown in Figure 9. For each gait parameter, a weight β is set. Suppose FM is the inertial feature vector output after the network model structure removes the last layer, F is the plantar feature vector output after the network model structure removes the last layer, and $F_S$ is the feature vector after splicing the two feature vectors. Then $F_S$ can be obtained by the following formula:

$F_s=\operatorname{conat}\left(F_M, F\right)\left[\begin{array}{c}F_M \\ F\end{array}\right]$             (2)

Next, the concatenated feature vector $F_S$ is assigned weights to obtain the fused feature vector $F_F$, which is calculated as follows:

$F_F=\beta F_S=\left[\begin{array}{c}\beta_M F_M \\ \beta_P F\end{array}\right]$             (3)

In Eq. (3), $\beta, \beta_M$, and $\beta_P$ represent weight vectors. After obtaining the fused feature vector, calculate the category score G:

$G= softmax\left(W F_F+b\right)$             (4)

In Eq. (4), $\mathrm{W} \in \mathrm{R}^{\mathrm{C}_1 * \mathrm{~F}_{\mathrm{s}}}$ and $\mathrm{b} \in \mathrm{R}^{\mathrm{C}_l}$, both are the weight matrix and bias vector respectively. Cl is the number of classification categories, and F_F is the dimension of the fused vector. The softmax function is defined as:

$softmax\left(Z_i\right)=\frac{e^{Z_i}}{\sum_{i=1}^{C_i} e^{Z_j}},\left(i, 1,2,3, \ldots C_l\right)$           (5)

9.png

Figure 9. Feature fusion based on adaptive weighting

Finally, the final predicted category y is calculated, and its calculation formula is:

$y=argmax(G)$             (6)

The loss calculation method in the feature fusion network model is the cross entropy loss function, which can be obtained by the following formula:

Loss $=\frac{1}{N} \sum_{\mathrm{j}} \operatorname{loss}_{\mathrm{j}}=-\frac{1}{\mathrm{~N}} \sum_{\mathrm{j}} \sum_{\mathrm{h}}^{\mathrm{C}_{\mathrm{l}}} \mathrm{y}_{\mathrm{jh}} \log \left(\mathrm{G}_{\mathrm{jh}}\right)$             (7)

The Eq. (7), N is the total number of samples, and Gjh represents the probability that sample j belongs to category h. yjh is a symbolic function, which can be expressed as:

$y_{jh}=\left\{\begin{array}{c}1(\text { The real category of sample } j \text { is } h) \\ 0(\text { The real category of sample } j \text { is not } h)\end{array}\right.$             (8)

In the model training phase, the single-modal ISA-ECA-CNN-BiLSTM model is first trained as a pre-training model. The multimodal adaptive weighted feature fusion model is then trained by fixing the network parameters of the ISA-ECA-CNN-BiLSTM model, and the weights in the feature fusion network model change with training. This training method can speed up model convergence and save training time.

In this study, we propose a fall risk assessment model based on the fusion of multisensory information to improve the efficiency of the channel attention mechanism of CNN-BILSTM. First, unlike existing fall risk assessment methods, our model accepts inputs as fixed-length segments, an improvement that allows the recognition model to be freed from noise- and bias-sensitive fall risk assessment tasks. Second, an improved network structure of an efficient channel attention mechanism is designed to incorporate a spatial attention module, which allows the model to efficiently extract global gait features. Based on this, based on this, gait features extracted from IMU signals and plantar pressure signals are adaptively weighted and fused, and the evaluation of our constructed dataset demonstrates that this approach outperforms existing fall risk assessment models and achieves new state-of-the-art performance in recognition and validation tasks.

This study had several limitations. First, the number of people in the constructed dataset is small and the size of the data is small. If it were trained using a larger dataset, the current model might perform better. Second, there were mainly male participants-75%). Future research should consider how male and female models perform differently. Deep learning models are also difficult to interpret. Future research that aims to create interpretable models may provide a more in-depth understanding of the factors that affect fall risk. Due to the lack of suitable public datasets, all experiments in this study were validated on a self-constructed dataset. In the future, it may be beneficial to consider employing external datasets for more comprehensive validation or cross-validation to ensure the generalizability of the model.