An Efficient Attention-Based LSTM Framework for Blood Glucose Level Prediction

Diabetes mellitus is a metabolic disorder that disrupts the body’s ability to regulate blood glucose (BG) levels. Beta cells in the pancreas secrete insulin, an endocrine hormone that regulates glucose utilization [1]. Type 1 diabetes is characterized by the loss of these insulin-producing beta cells, leading to insulin deficiency [2]. As a result, individuals with diabetes must continuously monitor their blood glucose levels, regulate them through consistent insulin administration, and make informed decisions about their medical regimen (e.g., once daily or before each meal) to maintain normal blood sugar level [3]. The primary objective in diabetes management is to optimize insulin dosages to prevent hyper/ hypo glycemia [4]. This is difficult due to the multitude of factors, including lifestyle, mental state, stress, and physical exercise, that also affect insulin intake, nutrition, and glucose levels [4]. Despite various advancements in diabetes observation, continuous glucose monitoring (CGM) remains aggressive. CGM devices are capable of providing an individual's glycemic condition at an assumed time. A positive recognition could dramatically expand the everyday behavior of diabetes by the patients themselves [5].

To simplify and accomplish the management of type 1 diabetes, a continual growth in automated procedures is quite essential, despite the advancements obtained thus far. In this way, commerce-precise BG devices might be game-altering [6]. These devices can provide early warnings about potential glycemic actions, enabling the implementation of either automatic or non-automated preventive measures. Moreover, these mechanisms are a criterion for the arrival of a closed-loop artificial pancreas as the present vision for the final automatic management of type 1 diabetes [7]. Hybrid modelling, data-driven techniques, and physical techniques are utilized for forecasting blood glucose levels. The data-driven method explores the present and historical values of diabetes management-linked variables to plan prospective BG excursions [8].

Generally, there are three primary categories of time-series prediction methods: classical time-series prediction, machine learning (ML), and deep learning (DL). The research reported various studies that used classical time series and ML based regression models, NNs, and DL methodologies for BG-level predictions. Eren-Oruklu et al. [9] designed an adaptive univariate autoregressive moving average (ARMA) model with fixed model orders solely based on CGM data. Meanwhile, Turksoy et al. [10] developed a multivariate ARMA with exogenous inputs (ARMAX) model, which used exogenous variables such as glucose and insulin while excluding meal information. Given the time-varying non-stationarity of CGM data, Yang et al. proposed an ARIMA model using an algorithm that adaptively determines model orders and simultaneously updates model parameters [4]. They found that their model outperformed the adaptive univariate and ARIMA models. As more research was done, Georga et al. [11] suggested SVR and SVR with feature selection using RF and RReliefF. Hamdi et al. [12] also proposed the DE-SVR for BG-level predictions.

Researchers found that ML-based classical time series struggle to predict BG dynamics due to their assumption of linear data, sensitivity to non-stationary CGM data, difficulty in capturing complex patterns, susceptibility to noise and outliers, and inability to adapt to changing conditions. However, even though SVR and SVR with kernel functions can handle non-linearity, the limitation of machine learning methods is how the input data is represented, which has an effect on how well the predictions work.

Motivated by the excellent modelling capabilities of artificial neural networks (ANNs) for nonlinear and non-stationary phenomena, several studies have employed ANN strategies to forecast BG levels. DL differs from traditional ANNs in the number of hidden layers, their interconnections, and their ability to abstract inputs. These intricate models with increased complexity outperformed conventional shallow NNs. These advanced DL based algorithms can autonomously discern patterns from data. While feed-forward ANNs fared better at predicting BG levels, traditional RNNs have discovery constraints due to the vanishing/exploding gradient problem, which is a major drawback of using these networks.

The incorporation of memory cells and forget gates into traditional RNNs has rectified this issue in Long Short-Term Memory (LSTM) networks.

Recent studies examined RNNs and LSTMs for enhanced accuracy in BG level prediction. Fox et al. [13] employed a RNN with Gated Recurrent Unit (GRU) cells, whereas Sun et al. [14] implemented a Bi-LSTM based RNN for BG prediction. Nevertheless, the precision attained by cutting-edge models for actual patient data is insufficient, rendering these methods potentially inapplicable for diabetes care. Conventional LSTMs sometimes forfeit essential information throughout extended sequences due to the restricted capacity of their memory cells to preserve data from distant time steps. The discovery of the attention mechanism allowed researchers to address long sequencing issues and concentrate on the most pertinent segments of the sequence. Traditional LSTM models may dilute sequential information over long time spans, potentially leading the model to overlook subtle yet influential glucose patterns. The attention method enables our model to concentrate on significant previous CGM readings that may more substantially impact future glucose levels.

This study offers an Optimal Attention-Based Long Short-Term Memory for BG Level Prediction (OALSTM-BGLP) Model. At first, it employs Min-Max Scaling to normalize the input data, certifying consistent and meaningful contrasts across dissimilar features. Furthermore, the OALSTM-BGLP approach utilizes the ALSTM technique, which includes an attention mechanism to selectively concentrate on related data within the input sequence while taking into account long-term dependencies. Besides, the method has been optimized by employing the RMSProp optimizer, which adjusts the rate of learning based on the magnitude of recent gradients, facilitating efficient training and convergence. The suggested methodology outperforms state-of-the-art methods in a performance evaluation conducted on the OhioT1DM dataset.

In this study, we have presented a new OALSTM-BGLP methodology. The foremost intention of the OALSTM-BGLP technique comprises different kinds of sub-processes involved in min-max normalization, ALSTM-based prediction process, and RMSProp optimizer-based parameter tuning. Figure 1 illustrates the workflow of the OALSTM-BGLP methodology.

1.png

Figure 1. Workflow of the proposed OALSTM-BGLP methodology

3.1 Min-max normalization

Firstly, the OALSTM-BGLP method employs Min-Max scaling to normalize the input. Min-Max normalization is a popular data conversion method to retain the sensitive features in the data [24]. By doing so, 8 weeks real time CGM data for six patients from OhioT1DM [25] is normalized by the use of min-max normalization based on consideration of maximum and minimum values in the data. This method carries out a linear conversion on the original data. It is especially suitable for the classification task and is used in several applications namely neural networks, artificial intelligence, clustering, nearest neighbor classification, and so on.

The goal is to normalize the original data D into a preserved form of data D’ that fulfills the privacy requirement with a least possible loss with high privacy information. This technique focuses mainly on data conversion through the min-max normalization to change the original data. All the attributes in a data are normalized by mounting the value such that they fall in the minimum range within [0.0-1.0]. V_i of attribute A from the range of [min_i-max_i] to [newmin_A-newmax_A].

$V_{i}^{'}=\frac{{{V}_{i}}mi{{n}_{A}}}{ma{{x}_{A}}-mi{{n}_{A}}}$(new$ma{{x}_{A}}$-new$mi{{n}_{A}}$)+new$mi{{n}_{A}}$           (1)

where, V_i indicates the new computed value. The relationship among the original values is preserved through the min-max normalization method.

3.2 ALSTM-based prediction process

For the prediction process, the OALSTM-BGLP technique uses the ALSTM model. In 1997, the LSTM network was initially developed [26]. Owing to its exclusive design framework, LSTM is appropriate for handling and forecasting significant actions with actual long intervals in time series. LSTM is nothing but a gated recursive neural network (GRNN) that inserts a gating device to switch the data spread in the NN based on RNN. For normal NN, the motive why performing gradient explosion is clarified below: there is a non-linear relation among the exterior condition at the latter stage and the exterior condition at this stage, and this link is assigned in every time step. The GRNN was able to resolve the problem of gradient explosion by raising the linear requirement among them. Generally, in LSTM, 3 control gate units contain namely output gate ${{o}_{t}}$ forget gate f_t and input gate i_t. The gates of forget and input are the bases for LSTM to recall the dependency of long-term.

Input gate (i_t) defines how much data about the existing state wants to be kept in the interior state:

${{i}_{t}}=\sigma \left( {{U}_{i}}{{h}_{t-1}}+{{W}_{i}}{{x}_{t}}+{{b}_{i}} \right)$           (2)

where, $\sigma$ denotes the function of logistics, $W_i$ and $U_i$ represent the weight matrix; $b_i$ refers to the bias term. $h_{t-1}$ refers to the output of memory block at time $t-1$ and $x_k$ signify the input vector at time $t$.

Forget gate (f_t) defines how much data from the previous wants to be rejected.

${{f}_{t}}=\sigma \left( {{U}_{f}}{{h}_{t-1}}+{{W}_{f}}{{x}_{t}}+{{b}_{f}} \right)$           (3)

where, $W_f$ and $U_f$ indicates the weight matrix; $b_f$ signifies the bias.

Output gate $\left(o_t\right)$ fixes how much data from the interior state wants to output to the exterior state at an existing stage.

${{o}_{t}}=\sigma \left( {{U}_{o}}{{h}_{t-1}}+{{W}_{o}}{{x}_{t}}+{{b}_{o}} \right)$           (4)

where, $W_0$ and $U_0$ denote the weight matrix; $b_0$ refers to the bias term.

The foremost steps of the LSTM prediction method are mentioned below.

(1) At first, the exterior state at the previous stage (h_t-1) and the input vector at the existing stage (x_t) have been employed to compute candidate state ($C_{t}^{\sim}$) and 3 gates;

(2) Next, merge the input gate (i_t) and forget gate (f_t) to upgrade the interior state (C_t) of the existing stage;

(3) Finally, dependent upon the output gate (o_t) the interior state data was transmitted to the exterior state (h_t).

The abovementioned steps can recognize the prediction of the LSTM technique recognized in Keras utilizing TensorFlow.

LSTM is capable of efficiently resolving the issue of gradient explosion [27]. At the procedure of training, overfitting will basis the test set to collapse. To resolve this issue, a dropout layer was inserted to enhance the framework of LSTM. Generally, the LSTM places the NN input vector to fixed measurement, which has reliable results while dealing with lower sizes. If the size of the input parameter is comparatively big, it will disturb the solution of the method owing to the size explosion issue. To concentrate on the prominent parameter, this research presents an attention mechanism (AM), which is recognized by preserving the LSTM intermediate output encoding for the input series.

Most attention techniques trust an Encoding‐Decoding structure [28]. The encoding procedure of the original codec method produces a vector of intermediate in the technique of Seq2Seq that is utilized to keep the data of the original series. However, the vector is fixed. If the input original series length is extensive, then this vector can able to store restricted data which bounds the accepting capability of the system. Use an AM to pause the restraints of the original codec technique on a fixed vector. Figure 2 represents the infrastructure of ALSTM.

2.png

Figure 2. Structure of ALSTM

The AM is primarily executed in the subsequent steps [26]. In LSTM, the output [h₁, h₂, …, h_n] is changed non-linearly to get [u₁, u₂, u₃, … u_n]. When predicting BG levels using CGM data with an Attention LSTM, more attention is typically given to the most relevant past glucose readings that strongly influence the current prediction. The AM dynamically allocates higher weights to time steps in the input sequence where patterns or trends are most critical for making accurate predictions.

The significance of each intermediate time step may be represented by the attention weight matrix α₁, α₂, α₃, …, α_n, which is generated by the AM. The weight matrix (W_k) can signify the prominence of the intermediate state. Lastly, a weighted sum has been executed for a parameter of input and weight to get the encoded vector $V$. The output y is acquired by decode as per the encode vector $V$. The complete formulation is mentioned below:

${{u}_{k}}=\text{tanh}\left( {{W}_{k}}{{h}_{k}}+{{b}_{k}} \right)$           (5)

${{\alpha }_{k}}=\frac{\text{exp }\!\!~\!\!\text{ }\left( u_{k}^{T}{{u}_{s}} \right)}{\mathop{\sum }_{k=1}^{n}\text{exp}\left( u_{k}^{T}{{u}_{s}} \right)}$           (6)

$V=\mathop{\sum }_{k}^{n}{{\alpha }_{k}}{{h}_{k}}$           (7)

The weight matrix is represented by W_k, a bias term or offset is denoted as b_k, normalised attention weights are represented by α_k, and the attention matrix for the time series CGM data is randomly initialised as u_s.

3.3 Hyperparameter tuning using RMSProp optimizer

Eventually, the parameter tuning process was executed by using the RMSProp optimizer to boost the ALSTM model performance. Optimizer is a hyperparameter required for training DL networks. They are widely classified into dual for non-convex optimizer issues, such as adaptive rate of learning optimization techniques, and non-adaptive rate of learning (classical SGD). Even though there is a second-order optimizer technique for a convex problem, the considerable variation among convex and non-convex optimizer issues for convex optimizer problem has one global goal, while the non-convex optimizer issue has more than one local goal. The objective is on first-order optimizer algorithms for non-convex problems. This is due to a non-convex optimization problem that is the most widespread in NN study. The loss surface defines the difficulty and complexity of finding the global optima. Therefore, an optimizer is employed for reducing the cost function which is evaluated by the cross-entropy, whereas the loss function calculates the error which shows the model efficiency. RMSprop is a gradient-based optimization method used in NN model training. A gradient of complex functions such as NN models tends to explode or vanish as the data transmits via the function. RMSprop was designed as a stochastic method for learning the mini-batch model.

RMSprop addresses the abovementioned problem through the moving average of the squared gradient for normalizing the gradients. This balances the step size (momentum), increases the step for a smaller gradient to prevent vanishing, and decreases the step for a larger gradient to prevent exploding. Geoffrey Hinton developed RMSProp, which is like an algorithm of gradient descent with motion [29]. RMSProp attempts to solve AdaGrad’s entirely reducing rate of learning by utilizing a squared gradient moving average that uses the scale of current gradient descent for standardization. So, with the rate of learning upsurge, this method utilized might travel in a straight way with faster converge of higher steps.

$E\left[ {{g}^{2}}{{]}_{t}}=0.9E \right[{{g}^{2}}{{]}_{t+1}}+0.1g_{t}^{2}$           (8)

${{\theta }_{t+1}}={{\theta }_{t}}-\frac{\eta }{\sqrt{\left( 1-\gamma  \right)g_{t-1}^{2}}+\gamma {{g}_{t}}+e}\cdot {{g}_{t}}$           (9)

where, g_t represents the moving average of squared gradients. γ denotes the term of decay which has a range from zero to one. In this work, the RMSProp optimizer is used to define the hyperparameter concerned in the ALSTM method. Below, we define the objective function that is used to measure the MSE.

$M S E=\frac{1}{T} \sum_{j=1}^L \sum_{i=1}^M\left(y_j^i-d_j^i\right)^2$           (10)

Here, M and L represent the resultant values of layer and data respectively, whereas $y_{j}^{i}$ and $d_{j}^{i}$ denote the attained and desired sizes for j^th unit from the resultant layer of the system at time t respectively.

The new OALSTM-BGLP model is designed for predicting BG levels by leveraging historical CGM data across four critical prediction intervals: 15, 30, 45, and 60 minutes. The selection of time intervals for BG level prediction is determined by clinical significance and the properties of the data. Glucose levels exhibit rapid fluctuations, particularly during mealtime. Shorter intervals, such as 15 minutes, are clinically significant for monitoring rapid glucose fluctuations, facilitating prompt interventions for insulin adjustments. While intermediate intervals (30–60 m) provide valuable insights into broader trends, they are beneficial for general glucose management, including meal planning, physical activity, and insulin administration. Additionally, the sampling frequency of CGM data (every 5 m) influences the selected PH, balancing model accuracy with real-time applicability. This approach ensures the model aligns with both clinical needs and data constraints.

Our results demonstrate that the proposed model outperformed various recent DL-based approaches in terms of predictive accuracy. While direct comparison with existing studies is feasible for the 30 m and 60 m predictions, a noticeable gap exists in the recent literature for 15 m and 45 m horizons. Table 6 illustrates its performance in comparison to new models tested on various versions of the OhioT1DM dataset.

Table 6. Comparison of recent literature utilizing the OhioT1DM Dataset with the OALSTM-BGLP model

Martinsson et al. [32] proposed an approach based on RNNs trained in an end-to-end fashion that required nothing but the glucose level history for the patient and predicted the BG on 30 m and 60 m PHs. The approach achieved the RMSE values of 22.43 and 36.39 for 30 and 60 m PHs, respectively. Falcone et al. used data from six patients from an updated version of the OhioT1DM dataset in their study to suggest a new deep residual time series forecasting architecture. This architecture modified N-BEATS by incorporating a bidirectional LSTM network into each block in place of fully connected layers. The method included supplementary input variables, including bolus insulin and carbohydrates, and introduced three additional loss terms. The proposed deep residual forecasting method achieved average RMSEs of 18.2 for the 30 m glucose forecasting interval and 31.7 for the 60 m interval. Li et al. [34] introduced GluNet, a glucose forecasting method, utilising data from six patients in the Ohio T1DM 2018 dataset and employing deep neural networks. The findings indicated that GluNet attained RMSE values between 19.28 mg/dl and 22.93 mg/dl for 30 m PH, whereas for 60 m PH, RMSE values varied from 31.83 mg/dl to 39.53 mg/dl. Freiburghaus et al. [35] introduced a multilayer CRNN that employs multivariate data from six patients in the OhioT1DM dataset. The RMSE obtained was 17.45 for a 30 m PH and 33.67 for a 60 m PH.

Zhang et al. [36] designed four different models, including deep learning algorithms and regression models, and tested how well they could predict BG levels within 30 or 60 m PH using data from 12 people with OhioT1DM. The two deep learning models are the DCNN and the Seq-to-Seq LSTM model. The two regression models are a MLR model and a BRC model. The Seq-to-Seq LSTM model exhibited superior performance in 30 m ahead predictions with a 17.52 Average. RMSE, whereas the MLR model excelled in 60m-ahead predictions with a 24.58 avg. RMSE. Zhu et al. [21] proposed a GRU-based RNN model, E3NN, that incorporates an attention mechanism and evidential regression. The model was developed and evaluated on three datasets, including OhioT1DM (12 subjects). The model scored RMSE 18.92 on 30 m PH and 32.54 on 60 m PH. Dudukcu et al. [30] proposed a fusion of LSTM, WaveNet, and GRU models. Experiments were done with data from 12 patients with OhioT1DM to make predictions on 30 m, 45 m, and 60 m PH, as well as 21.90, 29.12, and 35.10 Avg. RMSE was achieved, respectively. Yang et al. [37] developed an autonomous channel deep learning framework. Researchers tested the framework on data from 12 patients in the clinical OhioT1DM dataset. The study's experiments revealed an RMSE range of 16.775 to 21.085 on 30 m PH and 28.36 to 35.22 on 60 m PH.

In this study, we predict BG levels at 15, 30, 45, and 60 m into the future, offering a more granular and comprehensive analysis compared to most recent studies, which primarily focus on 30 and 60 m horizons. By including shorter (15 m) and intermediate (45 m) prediction intervals, our approach captures more immediate and nuanced glucose dynamics, which are critical for timely interventions and effective diabetes management. Our RMSE results demonstrate robust performance across all horizons.