Remote Patient Healthcare Monitoring and Pose Recognition over Video Sensors

Supervision of individuals in need of healthcare services is crucial for tracking their behavior and protecting their well-being, especially in caring for elderly people, rehabilitation centers, and hospitals. Given the fact that a rising number of people require constant medical supervision, the need for systems capable of creating permanent checks and responding immediately escalates. The present work proposes a health monitoring and surveillance system that incorporates multiple camera view analysis, employing machine learning and deep learning algorithms to assess the indices of physiology and behavior. Through early detection of health risks, such systems will greatly improve patient experience, particularly for one living alone, a critically injured patient, the elderly, or those in a rehabilitation center. Flashing light technology aims to enhance the quality of patient’s and senior’s lives due to constant activity and efficient emergency response. It reduces the pressure that healthcare providers face while delivering medical care and achieves the purpose of providing care at the right time. Apart from healthcare, the system uses are numerous; in workplace safety, one can look for hazardous behaviours in a bid to avoid accidents; in sports analytics, the system can monitor the physical and mental status of athletes and their performance.

HAR systems use multiple approaches to function, consisting of data collection followed by feature extraction and classification steps. Three main data acquisition instruments consist of RGB cameras and depth sensors together with wearable devices. Systems advance accuracy levels by processing different types of data inputs, including audio data and body signals. The traditional approach to feature extraction involved human-based methods that included Histogram of Oriented Gradients (HOG) and Scale-Invariant Feature Transform (SIFT), and spatiotemporal descriptors. The use of deep learning techniques, including Convolutional Neural Networks and Long Short-Term Memory and Transformer models, has gained wide acceptance in recent times. Sport vector machines (SVMs), together with deep neural networks, are among the classifier choices, along with data application-dependent network selection for these techniques, which need normalization or segmentation as preprocessing before classification.

The existing systems have achieved major breakthroughs, although they operate under essential boundaries. The computational requirements of deep learning models make them unsuitable for real-time healthcare usage, even though they are effective for their tasks. The development of models that generalize to real-world situations becomes challenging because existing systems require large labelled datasets that get collected primarily from controlled environments. Using visual cues exclusively delivers insufficient results because it overlooks crucial contextual or behavioral signals that would improve diagnostic accuracy plus timely reaction ability.

This motivates for development of a real-time context-aware and efficient framework. Researchers have highlighted an important void in the field because optimization-based methods, especially evolutionary algorithms, need integration with machine learning for developing weight-efficient systems. The current research has inadequate examination of how to unite behavioral and physiological inputs alongside multi-view visual data to create more comprehensive patient context.

To address these gaps, we suggest implementing evolutionary algorithms combined with machine learning principles into our architecture because they optimize feature selection, improve classification accuracy, and decrease computational load. The proposed system achieves successful classification of patient monitoring scenarios with notable precision, according to our results. Following this introduction, the paper combines the necessary content through these sections: Section 2 presents detailed research into current HAR systems along with their known weaknesses. The system design section explains the multi-view camera system and feature extraction protocol, along with details about classifier implementation. The evaluation of experimental outcomes and performance assessments is provided in depth in Section 4, and the paper concludes through Section 5 with a finding’s discussion.

The system aims to analyze input videos and recognize patient activities. Initially, the video is segmented into frames, which are then denoised by removing unnecessary background. An averaging filter is applied to blur the frames, followed by further denoising through the conversion of the RGB images into grayscale. After completing the denoising process, the frames are forwarded to the next step: human detection. Human detection is performed using two different methods: the first involves extracting human silhouettes, while the second uses pose estimation. Four distinct features are extracted from both detection techniques, two for each. For joint points, DOF, 3D Point Cloud features are extracted, while for full body Distance Transform, and GMM are utilized. These extracted features are optimized through early fusion and the Whale Optimization Algorithm (WOA). Then, for classification, we have employed a Maximum Entropy Markov Model (MEMM) classifier, which is applied to the proposed system and demonstrates superior performance compared. Figure 1 illustrates the general design of the proposed system.

1.png

Figure 1. Proposed architecture for classification

3.1 Image pre-processing

Preprocessing is a major input in many areas, including but not limited to machine learning, image processing, and deep learning. In this context, the original input is in the form of videos, and the actions associated with this input are segmented frames [7]. These frames normally come with background noise, which makes background subtraction an imperative step to get the object of interest to improve the quality of input data for other processes. Since image backgrounds and noise distract the algorithm from learning relevant features of the target object, the application of ML/DL increases the model’s efficiency and decreases the chance of over-fitting. Here, the process isolates pixels that are associated with the central object and removes unnecessary information from the background, thereby making the object of interest stand out. The other process is the smoothing process, which is used to remove detail, edges, or extra detail in an image. This is done by taking the mean of the pixel intensities in a neighborhood of each pixel, thereby resulting in an image with a smoother appearance. Reducing image noise has a great advantage in the subsequent calculations and reduces the amount of insignificant information. In image processing, ML, and AI applications, one filter type that we often use to perform smoother operations is the Gaussian blur filter. This filter then brings the center of the pixel into every image and blurs the surrounding area. In addition to this, conversion to grayscale is also common because working with 3D images will be more computationally costly [8]. A common type of this approach is quantization; it assigns a rational value between 0 and 255 to each of the pixels; in other words, it reduces a 3-dimensional picture to 2 dimensions. The outcome is the single-channel image view, which takes less perceptive power compared to the streams of the RGB model. The grayscale value is determined as a weighted average of the RGB pixel values using the values: Grey=0,2988R +0,5872G + 0,1137 B. Their weights provided concern about color intensity as being perceived by humans. These, as stated by NTPC (National Thermal Power Corporation Limited), are based on perception and could change under applications’ demands. Fine-tuned methods for the RGB-to-grayscale conversion do exist for varied different purposes. Figure 2 summarizes the pre-processing techniques in sequential order from top to bottom.

2.png

Figure 2. The steps of pre-processing are (a) Extracted Frame, (b) background removed, (c) blurred, (d) grey scaled

3.2 Human detection

Human detection is the determination of the presence and location of humans within a given frame or an image, much as a basic concept widely used in the many fields of computer vision [9]. This step is very important as it allows separate the object of our interest; thus, after enhancing procedures in the frame of our research, all the next processes will be concentrated on the identified human. To achieve this, we employ two distinct techniques: The main processes that should be mentioned are called image silhouette extraction and skeletonization. In silhouette extraction, we have a full body representation, while skeletonization paints the human figure with the barest minimum boundary as the essence of the figure isn’t the flesh but the bone. Specifically, for silhouette extraction, we use the Sobel operator, which is among the most widely used operators for edge detection. Edge detection techniques such as the Sobel operator compute the rate of change in intensity between two pixels or between two neighboring pixels, areas of high spatial frequency which are areas of edges. This is done using two 3×3 convolution kernels to identify changes in intensity in the x and y axes. Due to its simplicity and efficiency, the Sobel operator is highly effective in enhancing salient features while rejecting noise which is especially important when needs to detect edges accurately This step adds value to other subsequent processes such as feature extraction because it only focuses on areas of interest in an image thereby improving the efficiency of computation [10]. When used in conjunction with skeletonization, the use of the two models guarantees a rich data capture of human features that is vital for the subsequent analytic process as in Eq. (1) and Eq. (2). Figure 3 illustrates the Sobel kernels used for this operation, which can be described as follows.

$G_{x, y}(i, j)=\sum_{m=-1}^1 \sum_{n=-1}^1 G_{x, y}(m, n) \cdot I(i+m, j+n)$          (1)

where,

$G(i, j)=\sqrt{G_x(i, j)^2+G_x(i, j)^2}$          (2)

3.png

Figure 3. Binary images of human after extracting sobel image (a) NTU-RGB-D (b) ETRI-Activity3D, Illustrations of MOCAP (c) NTU-RGB-D (d) ETRI-Activity3D

These are the horizontal (x) and vertical (y) gradient kernels, respectively. For Sobel, these are typically 3×3 kernels, but for a more complex operator, you can increase to 5×5, is the summation now depends on a kernel of size (2k+1)×(2k+1), allowing for larger or more sophisticated filters, and $I(i+m, j+n)$ represents the pixel intensities in the neighborhood of pixel (i,j). Joint Motion Capture (MOCAP) is an analytical sub-component of the general motion capture technology that is more particularly related to the accurate direction and measurement of human joints or other articulated mechanisms. It is a paramount technology to applied domains such as biomechanics, sports science, computer animation and graphics, computer graphics, film and games, and virtual reality. What sets MOCAP apart is that while it delivers a detailed account of the operation of joints, it captures the highly complex motion of the skeletal structure that can accommodate complicated movements in physical modeling. The rotations at the joints are measured and modeled using quaternions, which offer an efficient and stable method of quantifying joint rotations. Eq. (3) and Eq. (4), present the quaternions that incorporate trigonometric functions and vector components that represent the axis of rotation and the angle of rotation equally well in a highly efficient method. Further, dual quaternions generalize this approach for including translations, which allow the modelling of combined rotational and translational joint movements, which is critical for effecting detailed capture and analysis of articulated systems.

$q(t)=\cos \left(\frac{\omega t}{2}\right)+\sin \left(\frac{\omega t}{2}\right)\left(u_x i+u_y j+u_y k\right)$          (3)

For combined joint transformations,

$२=q_r+\frac{\epsilon}{2} q_r \oplus(0, t)$           (4)

where, $q(t)$ is quaternion representing the rotation of a joint at a time, $\omega$ is the angular velocity of rotation, measured in radians per unit of time, $t$ is the time variable, indicating the specific moment at which the rotation is computed, $\left(u_x i+\right.$ $\left.u_y j+u_y k\right)$ is a unit vector that defines the axis of rotation in $(i, j, k) 3 \mathrm{D}$ space, ২ is dual quaternion representing a joint's combined rotation and translation, and $q_r$ rotational quaternion, describing the joint's orientation. The imagery output of human detection can be seen in Figure 3 where images a and b are Sobel images, and images c and d are MOCAP of humans.

3.3 Feature extraction

The first approach is based on the Degree of Freedom technique in which we find angles at the joint points and distances between them. This method also presents a new way of obtaining geometric features from human pose images by applying computer vision and data analysis. When using joint points as reference indicators, we calculate different geometric characteristics, including distances in distinct joint couples and angles in corresponding joints. This approach offers a more specific quantitative measurement of body postures and movements and all aspects of the biomechanics of human locomotion and provides a deeper insight into the subject. Furthermore, it increases the accuracy and robustness of pose estimation in various fields of interest, sports analytics, physical therapy, and human factors engineering. Eq. (5) and Eq. (6) use the Vector algebra geometry that brings together Euclidean distance and angle calculations to derive an all-encompassing range of features. This guarantees a solid quantitative context that facilitates understanding of human motion in two and three dimensions.

$F_{g e o}=\left\{\left\|J_i-J_j\right\|, \cos ^{-1}\left(\frac{\left(J_i-J_k\right) \cdot\left(J_j-J_k\right)}{\left\|J_i-J_k\right\| \cdot\left\|J_j-J_k\right\|}\right)\right\}$           (5)

Such that,

$i, j, k \in \mathcal{J}, i \neq j \neq k$            (6)

where, $J_i, J_j, J_k$ are the joint points in 2D or 3D space represented as vectors, $\left\|J_i-J_j\right\|$ is Euclidean distance between joints $i$, and $j,\left(J_i-J_k\right) \cdot\left(J_j-J_k\right)$ is dot product of vectors, $\mathcal{J}$ is the set of all detected joint points, and $F_{g e o}$ is set of all geometric features extracted, including distances and angles. Feature extraction from 3D point cloud is a crucial part in preparation of 3D spatial data for different uses in human motion analysis, object identification, identification of state of patient and 3D scene modeling. It includes procedures for defining and enumerating distinctive geometric and structural properties allowing the representation of intricate forms with low dimensionality [11]. Some of the identified techniques include Principal Component Analysis (PCA) which plays a role in reducing dimensions in the shape of the surface, curvature estimation for capturing the geometrical properties of the surface, and determination of the surface normal as a way of getting the orientation of points within the shape in the 3D space is also great way of characterizing the shapes in spatial sense. Recently developed potent PointNet++ employs deep learning for learning topological features, which drastically increases accuracy and provides robustness. PointNet++ is an extension of PointNet wherein we introduce hierarchical feature learning to address the issues of capturing not only the local relation of neighbors for point clouds and global features. This makes it very suitable for dense estimates from low-dimensional datasets, especially if the point density varies greatly. Moreover, 3D convolutional neural networks (CNNs) are used to improve feature extraction where the dependencies in the input domain using graph-based methods that can capture spatial dependencies and hierarchical structures in the input. The enhancement or fine-tuning of features really aids subsequent processes such as classification, segmentation, and registration, while improving recognition makes feature extraction a fundamental step in 3D data processing pipelines. Eq. (7) defines how PointNet++ aggregates feature hierarchically. At each layer, the local neighborhood is considered, and features are updated by combining spatial relationships with the features of neighboring points.

$\left.f^{(1+1)}\left(p_i\right)=\max _{p_i \in \mathcal{N}\left(p_i\right)} \varnothing\left(f^{(\mathrm{l})}\left(p_j\right)\right), \psi\left(p_j-p_i\right)\right)$           (7)

where, $\left(f^{(1)}\left(p_j\right)\right)$ is the feature vector of point $p_j$ at layer $l$, $f^{(1+1)}\left(p_i\right)$ is the updated feature vector of $p_j$ point at layer $l+$ 1, $\mathcal{N}\left(p_i\right)$ is the local neighborhood, $\psi\left(p_j-p_i\right)$ a positional encoding function to capture relative spatial relationships, and max is a symmetric aggregation function. Figure 4 is an imagery representation of the output of all joint point features.

4.png

Figure 4. Feature on human MOCAP (DOF)(a) NTU-RGB-D (b) ETRI-Activity3D, 3D point cloud (c) NTU-RGB-D (d) ETRI-Activity3D

The other three features that are extracted are BRISK, HOG and ORB. These feature points are applied to the full human body (silhouettes). BRISK (Binary Robust Invariant Scalable Keypoints) is a computer vision algorithm renowned for its robust feature detection and description capabilities. Utilizing binary patterns for image patch representation, BRISK excels in scenarios with varying lighting conditions and viewpoints, ensuring robustness and reliability. BRISK constructs a scale pyramid and detects key points using a variant of FAST, refining their positions for accuracy. It generates binary descriptors based on local intensity patterns around key points, facilitating efficient image matching and recognition in computer vision tasks. BRISK's binary descriptors enable efficient feature matching and resilience to noise, enhancing its usability in tasks such as the matching of images, recognition of an object, and reconstruction of 3D objects, where robustness, accuracy, speed, and working are paramount. Its further processing is explained in Eq. (8).

$H D(X, Y)=\sum_{i=1}^N x_i \oplus y_i=\sum_{i=1}^n\left(x_i, y_i\right)$           (8)

where, $b\left(x_i, y_i\right)$ denotes bit inequality, in Eq. (15), $x_i$ and $y_i$ are the $i^{t h}$ bits of the descriptors $X$ and $Y$ respectively. Its examples of both datasets can be seen in Figure 5. The last feature extraction technique is Gaussian Mixture Model (GMM) is a probabilistic generative model, which is considered to be a soft clustering model, much of the time used in feature extraction for clustering and density estimation. GMMs model data with a mixture of several Gaussian distributions, with parameters of mean vector and covariance matrix. To estimate the above-mentioned parameters of these Gaussian components from a given dataset, expectation maximization (EM) is one of the frequently used methods. This approach provides a probability to each and every datum belonging to each Gaussian component and thus highlights the distribution structure and variability of the datasets. Possible extracted features from GMM may be the posterior probability, mean, variance, as well as weight by each of the Gaussian components or a selection thereof. These features are useful in many machines learning processes, including classification, clustering, and anomaly detection, because they can represent high-dimensional multi-modal probability density functions. Eq. (9) explains the basic concept of GMM.

$\begin{gathered}p(x)=\sum_{k=1}^K \pi_k \cdot \frac{1}{(2 \pi)^{d / 2}\left|\sum_{k \cdot}\right|^{1 / 2}} \exp \\ \left(-\frac{1}{2}\left(x-\mu_k\right)^T \sum_k^{-1}\left(x-\mu_k\right)\right)\end{gathered}$           (9)

where, $p(x)$ is probability density function of the data point $x$, $K$ number of Gaussian components, $\pi_k$ is weight of the $K-$ th Gaussian component, where $\sum_{k=1}^K \pi_k=1, \mu_k$ is mean vector and $\sum_k$ th covariance matrix of the $K-t h$ Gaussian. Figure 5 shows the representation of full-body feature extraction points.

5.png

Figure 5. Feature on human full body (Distance transform) (a) NTU RGB-D (b) ETRI-Activity3D, GMM (c) NTU RGB-D (d) ETRI-Activity3D

3.4 Features fusion

Early fusion is a concept in machine learning and data processing where the diverse sources of information and features are combined early, commonly before input into a learning model or algorithm. In this approach, we combine the features from the different modalities or different extraction techniques into one. This fused representation is then introduced as data for the model [12]. The strong point of this approach is that all correlations and dependencies between features from different sources are combined right after the feature extraction step, allowing the final model to work with enhanced information. For instance, in human action recognition, features like DOF, and GMM could be concatenated into one vector retaining the separate contribution of each feature but improving the capability of the model to distinguish between features of complex patterns across different modalities. However, early fusion requires careful normalization and scaling of features to ensure compatibility and prevent any single modality from dominating the combined representation. Moreover, its working can be understood with the help of Eq. (10), and Eq. (11).

$y=f\left(\frac{\left[\sum_{i=1}^n \omega_i \cdot \emptyset_i\left(f_i\right)+V e c(H)-\mu_F\right]}{\sigma_F}\right)$          (10)

$H=\left(\sum_{i=1}^n \sum_{j=i+1}^n\left(\omega_i \cdot \emptyset_i\left(f_i\right)\right) \oplus\left(\omega_j \cdot \emptyset_j\left(f_j\right)\right)\right)$              (11)

where, $y$ is the final output, $f_i$ is the feature vector extracted by the different techniques, $n$ is the total number of feature extraction techniques, $\emptyset_i$ is the transformation function, $\omega_i$ are weight assigned to the feature vector, $F$ is the fused feature, $\mu_F$ is mean of values, $\sigma_F$ is standard deviation. Figure 6 shows the points distribution among different classes.

6.png

Figure 6. Graphical representation of feature fusion

3.5 Features optimization

In the context of machine learning, optimization is the process of setting the values of the parameters of a given model that best fit an objective function, usually error or loss. In performing learning, it assists in fine-tuning the weights and biases of the model in search of the required accuracy or efficiency. The Whale Optimization Algorithm (WOA) is an optimization method designed based on the hunting behavior of humpback whales and their bubble-net hunting strategy. The whale humpback whales optimization algorithm is a metaheuristic algorithm created to find the optimal solution to problems based on the feeding mechanism employed by humpback whales to exploit or explore the prey. In WOA, whales are modeled as candidate solutions within the search space. The algorithm alternates between two phases: exploitation and exploration. In exploitation, the whales advance towards a current solution, thus forming a spiral or encircling motion reminiscent of bubble-net hunting. In exploration, whales randomly search the space by moving far from the current best solution to avoid local optima. The balance of these phases is regulated by a parameter that is reduced linearly across iterations. These behaviors, in sequence, enable WOA to locate the global optima of numerous challenging optimization issues. Eq. (12) is the mathematical representation of WOA.

$X_i^{(t=1)}= \begin{cases}X^*-A \cdot D, & \text { if } p<0.5 \text { and }|A| \leq 1 \\ X^*+D^{\prime} \cdot e^{b l} \cdot \cos (2 \pi l), & \text { if } p<0.5 \\ X_{\text {rand }}-A \cdot D, & \text { if }|A|>1\end{cases}$            (12)

where, $X^*$ Represents the position of the current best solution (i.e., the leader whale), $X_{\text {rand}}$ is a randomly chosen solution from the population used for exploration, $A$ is coefficient vector that controls the convergence behavior, $D$ is the distance between a whale's position and the best solution, $D^{\prime}$ modified distance vector for spiral motion, $p$ is random number in $[0,1]$ that determines whether to exploit (closer search) or explore (global search) the solution space, $b$ is constant defining the shape of the spiral for encircling prey, $l$ is a random number used in the spiral equation for diversification, $e^{b l}$ an exponential decay factor controlling the amplitude of the spiral movement. Algorithm 1 gives full implementation of the optimizer. Figure 7 shows a 3D graph representation of WOA.

7.png

Figure 7. Accuracy optimization graph using whale optimization

3.6 Events classifier: MEMM

A MEMM classifier is a Maximum Entropy Markov Model that is a sequence model used to label as well as to segment sequences, such as in natural language processing and time series analysis. It integrates two concepts, the maximum entropy, a probability distribution theory that maximizes entropy, and the function of Markov, which holds that the conditions currently rely upon the previous condition. MEMMs estimate the probability of the next label when the current observation has been made, and the previous label known. This is done by using a conditional probability distribution commonly estimated by using logistic regression. The inputs to the model are feature sets; these are vectors that provide context as to the sequence: the model then estimates the probability by using the principle of Maximum Entropy; that is the model aims at choosing the least biased probability distribution for the data given certain constraints.

Maximum Entropy Markov Models (MEMMs) effectively solve problems that depend on sequence order or time-based events. The predictive process for MEMMs depends on both present input data and past label information. A logistic regression model estimated the necessary probabilities by using refined feature vectors. L2 regularization came into use during training as an effective measure against overfitting, which made the model more successful at generalization. The model required optimal configurations, which were discovered through a grid-searching process. The team evaluated multiple values of regularization strength alongside window dimensions and past information weight as they related to the Markov assumption. Through cross-validation, the model values achieved their final configuration because they demonstrated consistent performance across distinct data partitions.

During the prediction stage, the Viterbi decoding algorithm was applied to determine the most likely sequence of labels, using the conditional probabilities generated by the MEMM. This approach guarantees that the output sequence is the best possible one overall, considering the full context of the input. The complete process, starting from feature extraction, combining and refining the features, and leading up to classification with the MEMM, was carefully structured to ensure consistent results and strong performance across different runs. Algorithm 2 gives a complete pseudo-code for MEMM. Figure 8 shows the functional architecture.

8.png

Figure 8. Functional structure of MEMM classifier

A method which unites feature detection algorithms with optimal classifier systems enables the detection of key events during patient gait measurement. Mathematical models in the system track successive human body movements to perform precise behavioral analysis and event identification. The method achieves strong benchmark results. Upcoming work aims at four main enhancements for better practical use and improved stability. The next stage of development involves making the model more efficient for lower power consumption purposes and real-time edge-device capability alongside integration of biomechanical data elements such as joint motion measurement and energy expense metrics. We planned to add wearable sensor data as a supplement to visual information for enhancing recognition performance when vision is obstructed or when lighting is poor. The model development extends to analyze aerial video obtained from drones, which will permit the study of group characteristics along with event identification across extensive public areas.

As the video-based patient monitoring systems create ethical and privacy issues. This research employs only anonymized public benchmark datasets (NTU RGB+D and ETRI-Activity3D) that meet approved consent standards; however, we understand that deployment requirements in clinical or residential settings demand strict adherence to privacy rules and ethical practices. The research demands obtaining patient consent along with strong data anonymization practices combined with secure databases and processing systems. The development of edge computing alongside federated learning and pose-based abstraction solutions aims to improve user privacy protection capabilities in future research.