GeriatricCare 4.0: A Novel 3D Context-Based CareVision Framework for Fall Detection, Fall Classification and Fall Alerts for Elderlies

A recent report on aging and health predicts that by 2050, there will be twice as many senior citizens over 60 outnumbering other age groups [1, 2]. As a result, concerns about the health and safety of the elderly are mounting. Falling is one of the factors that contributes to fatalities and serious injuries in the elderly. An immediate approach to elderly patients may help keep them out of danger or perhaps save their lives. Therefore, it’s essential to offer senior citizens medical aid following a fall in order to protect their health. Whether it comes to geriatric care or people with medical disorders that make them prone to falling, fall detection technology is used to automatically detect whether a person has fallen [3]. The primary objective of fall detection systems is to offer early help to the individual, such as alerting family members, caretakers, or emergency services, in order to ensure prompt medical attention and avoid any potential harmful fall-related effects [4, 5].

Figure 1 represents classification of fall detection approaches. As shown in Figure 1, data from human actions is used in approaches to sensor-based statistical falls, either in the form of inertial sensors or other sorts of sensors. The term "vision-based fall detection" refers to a technique for detecting and classifying fall positions that watches people’s movements with cameras or other visual sensors. In the presented research work, we have aimed at elderly fall detection and forecasting using a customized 3D CareVision Framework. Falls can have serious effects and necessitate rapid medical attention. Vision-based fall detection systems may automatically recognize and respond to probable falls by utilizing computer vision algorithms, machine learning, and image processing techniques [6]. Anitha and fellow researchers have proposed a visual fall recognition system using an LSTM based fall detection approach using human skeleton key points. However, they did not propose a complete 3D elderly care vision framework for fall detection and forecasting [7]. Amsaprabhaa and fellow researchers have implemented a fused vision based fall detection approach using spatiotemporal data and skeletal gait features. However, they did not propose 3D vision-based fall detection, fall forecasting, and fall alert approaches. Furthermore, they did not discuss any ideas related to aging care, fall alert systems [8]. Wang and his team have discussed ideas related to human motion recognition using a triboelectric nanogenerator. However, they did not discuss any ideas related to vision-based elderly fall detection [9]. Quan and fellow researchers have proposed vision based tactile sensing for large data sets. However, they did not discuss or propose any frameworks for fall detection and forecasting [10]. Lian and his team have proposed a video surveillance based approach for fall detection. However, they did not discuss any ideas related to 3D vision-based fall detection methodologies [11]. Zhang and fellow researchers [12] have discussed image-processing based fall detection in bus compartments. However, they did not discuss anything related to elderly care, elderly fall detection, and fall alerts. Pian and his research team have proposed a multimodal approach for fall detection. The approach was focused on achieving data privacy and security for the fall detection data. However, they did not discuss any ideas related to elderly falls, fall alerts, fall detection, or forecasting [13]. Wang and fellow researchers have implemented CNN-based fall detection using radar signals. However, they did not discuss anything related to elderly fall detection, or aging care [14]. Li and his team have discussed a frame-prediction based fall detection approach using pre-captured surveillance video samples. However, they did not discuss any ideas related to 3D-vision based fall detection for elderlies [15]. De and fellow researchers have proposed an indoor body activity based classification approach for fall detection [16]. However, they did not discuss any ideas related to elderly care, and elderly fall alerts. Mobsite and her team [17] have proposed posture based fall detection system for monitoring human activities. However, they did not discuss any ideas related to vision based fall detection and forecasting, and elderly care. Wang and fellow researchers have performed a detailed comparative analysis of various AI methodologies for fall detection of human movements. However, the researchers [18] did not discuss any ideas related to vision based fall detection, IoT enabled fall detection and forecasting, or monitoring elderly people for emergency situations. Sofia and her fellow researchers have proposed an SVM-enabled fall detection methodology for wheelchair based fall detection. However, they did not discuss any ideas related to elderly care, vision-based fall detection and forecasting [19]. Wu and fellow researchers [20] have proposed a floor vibration based wireless fall detection methodology. However, they did not discuss any ideas related to vision based fall detection and forecasting, and elderly care. Li and his research team have discussed a time-series data based fall detection approach using key points. The approach was focused on achieving data privacy and security of the fall detection data. However, they did not discuss any ideas related to elderly falls, fall alerts, fall detection [21]. Yousuf and his fellow researchers [22] have proposed a ubiquitous sensor-based architecture for fall detection. However, they did not discuss any ideas related to elderly care, vision based fall detection. Rezaei and fellow researchers [23] have proposed a human fall detection approach using radar sensors. However, the researchers did not discuss any ideas related to vision based fall detection and forecasting. Mojidra and fellow team members [24] have proposed an AI-based vision approach for crack detection and forecasting. However, they did not discuss any ideas related to vision-based fall detection and forecasting, and elderly care.

1.png

Figure 1. A representation of various ambient-based, wearable-based and vision-based fall detection methodologies

In this study, we have proposed a novel 3D Carevision Framework for analyzing and evaluating elderly fall detection movements in various directions and notifying family members in emergency situations. In conducted experiments, we have discussed a customized iterative conformal geometric algebra methodology, a customized 3D Open Pose-based human skeleton detection, and an AI-enabled CareVision Posture Classification Framework for fall detection and forecasting. In recent times, fellow researchers have carried out scatter attempts and proposed vision-based fall detection methodologies [25-34]. However, a complete 3D CareVision framework for fall detection, fall alerts, and forecasting for elderlies has remained an open research problem for fellow researchers.

1.1 Objectives and contributions

The primary objectives and major contributions of the proposed 3D CareVision Framework are the following:

• The elderly video dataset creation termed "Geriatric-2000", along with the benchmark Le2i fall video sequences dataset. In the presented research work, we have applied the widely known benchmark dataset "L2ei" and the customized "Geriatric-2000" dataset, which contains more than 6,000 elderly fall video sequences. The 25 human skeleton features are extracted using the customized 3D Open Pose methodology.
• Analysis and Detection of twenty-five human skeleton points using the customized 3D Open Pose Methodology have been discussed as described in section 2.
• Design and Development of a Customized 3D Open Pose CareVision Fall Estimation and Detection Framework for Fall Detection of Elderlies.
• Design and Development of a Customized LSTM-CNN Enabled Context-based 3D CareVision Framework for Fall position classifications.
• The proposed 3D CareVision Framework will assist elderly personnels in case of emergencies and notify house members by sending emergency fall alerts.

Multi-person skeleton information can be detected in real-time by the 3D OpenPose human key node recognition system. It employs the feature vector affinity parameter to determine the hot spot map of human key nodes after using the top-down human body attitude estimate technique to identify the locations of key points on the human body. OpenPose is capable of realizing posture estimate as well as human movement, facial expression, and finger movement. It has outstanding robustness and is appropriate for both one person and several people. The crucial point coordinates are extracted and normalized after the posture estimation method tracks and outputs the key point coordinates of the human 3D skeleton [25, 26]. H={h₀, h₁, · · · , h₁₃} denotes the joint position set for ease of representation. The Joint Coordinates ( hc ) are defined as: Establish the node j’s position at time t as h_c(t)=(x_tn, y_tn), n∈{0, 1, · · · , 13}. In conducted experiments, the surveillance camera enables 3D OpenPose CareVision Estimation methodology to gather data on twenty-five human key nodes. Figure 3 represents a processing of an elderly frame using the 3D human skeleton model.

3.png

Figure 3. Elderly video frame processing using 3D Human skeleton model

4.png

Figure 4. Video sequences of elderly falling process using 3D CareVision estimation and fall detection methodology

The center of gravity of the human body will shift vertically during a quick fall, as depicted in Figure 4. The center of the hip joint can mirror this feature and serve as a representation of the body’s center of gravity. The longitudinal coordinates of the hip joint center point of each frame of the image are determined by processing the joint point data from the Open Pose. It is identified once every five adjacent frames with a time interval of 0.25 seconds because the transition from standing posture to falling posture takes place in a very brief amount of time.

Hips coordinates can be defined as: $\mathrm{h} c_1 \mathrm{o}(\mathrm{n})=\left(\mathrm{p}_{\mathrm{n} 10}, \mathrm{q}_{\mathrm{n} 10}\right)$ and $\mathrm{x}_{10}(\mathrm{t})=\left(\mathrm{x}_{\mathrm{n} 10}, \mathrm{y}_{\mathrm{n} 10}\right)$. The center of the hip coordinates at the time $\mathrm{n} 1$ is $q n_1=\frac{q_{n 1} 10+q_{n 1} 10}{2}$ and the $\mathrm{y}$-coordinate at time $\mathrm{n} 2$ is $q n_2=\frac{q_{n 2} 10+q_{n 2} 10}{2}$. The hip center descent velocity can be defined as:

$\Delta_n=n_2-n_1$          (1)

$\mathrm{dv}=\frac{\left|q_{n 2}-q_{n 1}\right|}{\Delta n}$       (2)

where, the value of z is greater than or equal to the descent velocity $\overline{d v}$, the elderly fall is detected. According to the conducted experiments, we have set 0.0085 p/q as the fall threshold of the falling velocity of the detected center of the hip joint of an individual, where $\mathrm{z} \geq \underline{\mathrm{z}} X_1=1$.

$X_1=\left\{\begin{array}{l}0 ; z<\overline{d v} \\ 1 ; z \geq \overline{d v}\end{array}\right.$      (3)

The human body’s tendency to tilt is the most noticeable aspect of falling, and this tendency will continue to increase. In this study, a human centerline HC is created in order to express the characteristics of the body’s constant tilt during the process of human fall (Let l be the link between midpoint mp and joint mp0, and let mp12 be the midway of joint mp12 and joint point mp25). Where angle between the ground and the human centerline is defined as θ.

The proposed customized 3D Open pose methodology detects the joint points $0,1,2, \ldots 25$ are $m p_0(t)=\left(p_n, q_n\right)$, $m p_{10}(\mathrm{t})=\left(\mathrm{p}_{\mathrm{n} 10}, \mathrm{q}_{\mathrm{n} 10}\right)$ and $\mathrm{mp}_{13}(\mathrm{n})=\left(\mathrm{p}_{\mathrm{q} 13}, \mathrm{q}_{\mathrm{n} 13}\right)$ respectively. Therefore, $\overline{m p}=\frac{m p_{10}+m p_{13}}{2}, \overline{m p}(\mathrm{n})=\left(p_n, q_n\right)$. The angle between the ground and the human centerline at time $\mathrm{n}$ can be defined as,

$\theta_n=\arctan \left|\frac{q_{n 0}-\overline{q_n}}{p_{n 0}-\overline{p_n}}\right|$       (4)

$X_2=\left\{\begin{array}{l}0 ; \theta \geq \theta_0 \\ 1 ; \theta<\theta_0\end{array}\right.$       (5)

when θ<θ₀ (θ₀=45◦) M₂=1, can be considered as the occurrence of an elderly fall event. The body tilt, which becomes more pronounced when one is falling, is the most noticeable aspect of a human body. The center line HC of the human body is established in this study to reflect the characteristics of the body’s continuous tilt during the process of human fall (let the link between the midpoint of joint mp and joint mp0 be g, and let the midpoint of joint mp12 and joint point mp13 be h). The outer rectangle of the human body has a width to height ratio of HP=Width/Height. The outer rectangle of the target will change as the human body hits it; the change in the length-to-height ratio is an important parameter in this situation.

$X_3=\left\{\begin{array}{l}1 ; H P \geq N \\ 0, H P<N\end{array}\right.$       (6)

where, the threshold value is N. The width-to-height ratio HP of an elderly body when walking properly is less than 1, whereas it is more than 1 when falling occurs, according to the elderly fall occurrence event. The occurrence of the elderly fall event is satisfied when,

$\mathrm{HP} \geq \mathrm{N}, X_3=1$       (7)

5.png

Figure 5. Elderly standing process after fall using 3D

CareVision Estimation and Fall Detection methodology

No fall alert will be generated if an elderly person can stand up on their own after falling. The majority of fall detection relies on the examination of the fall process, without considering the fact that most people rise up on their own shortly after the occurrence of a fall event. As depicted in Figure 5, getting up after falling might be seen as the opposite of falling. The slowness of the entire procedure compared to a fall is the only distinction. Algo. 1 describes the step-wise 3D Open Pose CareVision Fall Estimation and Detection Methodology.

In conducted experiments, we have applied elderly video sequences, generated the corresponding 25 3D human skeleton joint points, and trained the proposed 3D CareVision LSTM-CNN Enabled Framework for fall detection and fall position classification of elderly as described in Section 4.2. The customized LSTM-CNN enabled deep learning model is compared with the customized machine learning methodologies such as Fine KNN (K-Nearest Neighbors Algorithm), Medium KNN, Decision Tree, and customized deep learning approaches such as artificial neural network (ANN), long short-term memory network (LSTM), and Recurrent Neural Network (RNN), which have received corresponding accuracy of 84.21, 85.09, 87, 89.91, 90.86, 91.5, and 92.4. The customized 3D CareVision LSTM-CNN-enabled Framework for fall detection and fall position classification Framework has achieved an accuracy of 98.23 percent and a ROC value of 0.96. Figure 7 and Figure 8 depict the training and testing accuracy and loss representation of the proposed 3D Context-based LSTM-CNN-enabled Fall Detection and Fall Classification approach. Figure 9 represents the ROC curve representation of the proposed CareVision, and the customized machine learning methodologies such as Fine KNN (K-Nearest Neighbors Algorithm), Medium KNN, Decision Tree, and customized deep learning approaches such as artificial neural network (ANN), long short-term memory network (LSTM), and Recurrent Neural Network (RNN). Figure 10, Figure 11, Figure 12, Figure 13 represents the fall position classification results of the proposed 3D Context-based LSTM-CNN-enabled Fall Detection and Fall classification approach in standing, sitting, bending, and lying positions. Table 1 describes the metrics comparison of the customized CareVision LSTM-CNN framework, with the customized machine learning methodologies such as Fine KNN (K-Nearest Neighbors Algorithm), Medium KNN, Decision Tree, and customized deep learning approaches such as artificial neural network (ANN), long short-term memory network (LSTM), and Recurrent Neural Network (RNN).

7.png

Figure 7. Training and validation accuracy representation of the proposed 3D CareVision LSTM-CNN enabled framework

8.png

Figure 8. Training and validation loss representation of the proposed 3D CareVision LSTM-CNN enabled framework

9.png

Figure 9. ROC curve representation of proposed 3D CareVision LSTM-CNN enabled framework

10.png

Figure 10. Fall classification results in a standing position

11.png

Figure 11. Fall classification results in a sitting position

12.png

Figure 12. Fall classification results in a bending position

13.png

Figure 13. Fall classification results in a lying position