Modelling of IoT Signal Analytics and Feature-Aware Transformer Architecture for Early Diabetes Prediction in Smart Healthcare

Figure 1 presents the architecture overview of the proposed ISA-FT model. Healthcare data have been first collected and preprocessed through steps like cleaning, encoding, outlier detection, and normalization. Second, a feature selection approach such as Boruta is adopted for filtering out beneficial features of the dataset. Following this step, the selected feature set can be further visualized through SHAP to evaluate the features' importance for classification. Ultimately, the filtered feature dataset is sent as the input to the FT-Transformer to capture the intricate interdependencies between features via self-attention. Binary classification is performed to classify between non-diabetic and diabetic patients by employing an MLP classification head. The optimization algorithm AdamW has been utilized during the training stage to efficiently guide the learning process and enhance the model's generalization capabilities.

Figure 1. Overall architecture of the IoT Signal Analytics based Feature-Aware Transformer (ISA-FT) model

3.1 Dataset description

For implementing the proposed approach, we utilized the diabetes prediction in India Dataset available at Kaggle [18].

The diabetes prediction in India dataset was chosen for this study due to its extensive database of demographic, lifestyle, physiological, and clinical information, which closely mimics the diverse data produced in IoT-powered healthcare settings. In contrast, the selected dataset has more samples (5,292 instances) and more healthcare variables, which facilitate learning feature interactions more effectively with the proposed FT-Transformer architecture than conventional datasets like the PIMA Indian Diabetes Data Set. Moreover, the presence of several health-related attributes allows meaningful feature selection using Boruta and model interpretability analysis by SHAP. This dataset contains a total of 5,292 observations with 27 original features related to lifestyle, physical, and clinical conditions. Among them, 18 key features have been identified after feature selection to develop the model. The 27 original attributes of healthcare data were examined, with nine that proved less informative or redundant dropped and 18 identified as being highly relevant by Boruta. The feature subset was then used to perform SHAP analysis and quantify the contribution of each feature to the prediction output. Instead of a second feature elimination step, SHAP's results are validated by ranking the features based on their global importance and giving explanations for each prediction at the local level. The extracted SHAP ranking was similar to the Boruta-based feature subset, which confirmed that the importance of the features was not significantly influenced by the proposed prediction system. It is a marginally balanced dataset with 2,588 observations representing the non-diabetic individuals and 2,704 observations representing the diabetic individuals, as illustrated in Table 2.

Table 2. Dataset description and class distribution of the diabetes prediction in India dataset

A marginally balanced distribution helps to alleviate the sampling-based biases in model learning, as well as improve the stability and uniformity of the classification performance. For adapting the study in the realm of the IoT-enabled healthcare ecosystem, each feature has been envisioned corresponding to some relevant wearable or biosensing device for generating real time physiological signals, as depicted in Table 3.

Table 3. Mapping of dataset attributes to corresponding IoT-based wearable and biosensing devices

In real world IoT healthcare settings, physiological sensor readings can be subject to motion artefact, communication delay, environmental noise, and sometimes signal loss. The use of an offline benchmark dataset does not make the present study less applicable to real-world data; the proposed preprocessing stage (outlier removal and normalization) does offer some robustness against noisy measurements.

3.2 Preprocessing

The preprocessing stage is performed to improve data quality and enhance the effectiveness of the diabetes prediction model [19]. Initially, the dataset will be encoded, outliers will be removed & discarded, and MinMax normalization will be applied to all features before applying feature selection and classification.

•Encoding: The dataset consists of numerical and categorical features, such as 'Physical activity', 'stress level', and 'hypertension', which is a categorical feature, so we transform it to a numerical value using the appropriate encoding technique so that the FT-Transformer model is capable of handling this heterogeneity.

•Outlier detection and removal: Outliers can adversely impact model performance because outliers introduce new and extreme features into the dataset, which makes the prediction process less accurate. Hence, outlier detection and elimination are required in order to clear away these extreme values for a good diabetes prediction dataset.

•Min-max normalization: After outlier removal, the feature values are normalized using Min-Max normalization to scale all attributes within a common range. The Min-Max normalization is computed as:

$X_{\text {norm }}=\frac{X-X_{\text {min }}}{X_{\text {max }}-X_{\text {min }}}$    (1)

where, $X_{\text {norm}}$ is the normalized feature value, $X$ represents the original feature value, $X_{\text {min}}$ indicates the minimum value of the feature, and $X_{\text {max}}$ is the feature's maximum value Encoding, outlier detection and removal, and Min-Max scaling process the dataset to produce a unified representation, setting up the Boruta-SHAP feature selection and FT-Transformer diabetes prediction tasks for enhanced outcomes.

3.3 Boruta-SHapley Additive exPlanations feature selection

This section explains the feature selection process used for identifying the most relevant features and improving model interpretability.

3.3.1 Boruta

The intent behind this work is to figure out which are the important features to determine for early predicting diabetes using the Boruta algorithm [20]. The Boruta algorithm is the feature selection algorithms that use RF. Boruta identifies a feature by comparing attributes against features with the permutation of the features of those features known as "shadow features". They make shadow features by permuting the current feature set of the features, creating a new feature set, and running the model with both datasets to get a measure of the features' importance. Boruta tries to measure the relative importance of current features against shadow features using the Z-score, such that they determine whether actual features are more significant for making predictions and help them to find the more significant features. The Boruta algorithm generates shadow features by randomly shuffling the original feature values and trains an RF model using both original and shadow features. Feature importance scores are then computed and compared using Z-scores, where features that consistently achieve higher importance than their shadow counterparts are retained as relevant, while less significant features are discarded.

$Z_i=\frac{I_i-\mu_{\text {shadow }}}{\sigma_{\text {shadow }}}$     (2)

where, $I_i=$ Importance score of the $i^{t h}$ feature, $\mu_{\text {shadow}}=$ Mean importance score of the shadow features, and $\sigma_{\text {shadow}}=$ Standard deviation of the shadow feature importance scores. The Boruta algorithm has a unique advantage when it comes to selecting optimal features since it includes all pertinent features, even those features that have interactions with other features. The method aims to keep important predictive features to prevent overfitting and maintain the generalization performance of the system. In this present study, the Boruta algorithm is used to discard non-relevant information provided by IoT-based sensors to make a decision for predicting the outcome of diseases that influence diabetes patients.

3.3.2 SHapley Additive exPlanations

While Boruta excels in selecting a portion of informative features from the original feature set by comparing the original with generated shadow features (for noise and redundancy reduction), the approach itself acts like a feature filter. To serve this purpose and to further enrich post-hoc explanation for features, we then apply SHAP for the selection and ranking of those important IoT features. The two combined, i.e., Boruta first for selection of the IoT features, followed by SHAP for ordering and interpretation, offer a two-tier solution where important features are screened by the latter, and the interpretability aspect for the prediction of diabetes based on this feature set is analyzed by the FT-Transformer.

By providing an explanation, SHAP reports the influence of each feature in a prediction while averaging over all possible combinations. It works by evaluating the marginal contribution of every feature against all other combinations to provide model-wise or local explanations without ignoring interactions. SHAP overcomes some of the limitations in traditional model interpretation methods, for example, by providing model-agnostic interpretation and allocating contributions fairly according to cooperative game theory. Thus, this algorithm makes SHAP an applicable method in healthcare domains. The SHAP value is calculated as:

$\phi_i=\sum_{S \subseteq F \backslash\{i\}} \frac{|S|!(|F|-|S|-1)!}{|F|!}[f(S \cup\{i\})-f(S)]$  (3)

Here, $\phi_{\mathrm{i}}$ represents the contribution of feature $i$ to the prediction, and the remaining terms define the feature subsets and model outputs involved in SHAP-based importance estimation.

3.4 Feature Tokenizer Transformer with Multi-Layer Perceptron classification head

This section presents the FT-Transformer model with an MLP classification head used for diabetes prediction. The FT-Transformer model with an MLP classification head is designed to address the drawbacks noticed in standard ML approaches when performing on a combined diabetes dataset [21]. However, the FT-Transformer successfully conducts feature interactions by multi-head self-attention (MHSA) and token embeddings; its ability to convert the learnable representations into precise prediction outcomes needs a better classification mechanism. Hence, an MLP classification head is incorporated, and then the transformer encoder is utilized to increase the discrimination ability. This framework allows for gaining global contextual dependencies using attention, while executing efficient non-linear classification via dense transformations. The FT-Transformer architecture obtains the feature-selected and preprocessed input matrix $X_{\text {train}}$. The feature tokenizer can be utilized to combine all the feature columns as a learnable vector representation. Considered an input vector $x \in \mathbb{R}^d$, the tokenizer maps to every feature $x_j$ and token embedding as given below:

$t_j=E_j x_j+b_j$     (4)

Here, the parameter $t_j$ characterizes a $j^{\text {th}}$ token embedding, and the mathematical form of $E_j \in \mathbb{R}^{d_{\text {model}}}$ represents the feature-related projection matrix. Later, during tokenization, the positional encodings have been integrated into the token sequences and entered into the transformer encoder module. In the encoder, MHSA calculates contextual correlations between the embedded features:

$\operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$   (5)

In this mathematical form, $d_k$ points to the dimension of the key vectors, $K$ indicates the key matrix, $V$ denotes the value matrix, and the term $Q$ indicates the query matrix.

This method allows the model to dynamically obtain feature interactions between IoT sensor-based features, namely fasting blood sugar, physical activity, stress level, BMI, postprandial blood sugar, and hypertension, thus gaining intricate dependencies related to diabetes risk. Additionally, the outcome of every self-attention layer is handled by residual connections, layer normalization (LN), and a feed-forward transformer module to produce the higher-level contextual representation represented by $H_{F T}$.

Later, the produced transformer representation is entered into an MLP classification head that contains fully connected (FC) layers with non-linear activation functions. The MLP is calculated by:

$\begin{aligned} & Z^{(1)}=\sigma\left(W_1 H_{F T}+b_1\right) \\ & Z^{(2)}=\sigma\left(W_2 Z^{(1)}+b_2\right)\end{aligned}$ (6)

Here, the term $b_1$ and $b_2$ represent bias vectors, the parameter $W_1$ and $W_2$ indicate trainable weight matrices, and the symbol $\sigma(\cdot)$ points out the nonlinear activation function like GELU.

The MLP classification head improves the discriminative ability of the transformer representation through learning intricate nonlinear decision boundaries related to diabetes prediction. The final prediction representation can be determined by,

$u=z^{(2)}$    (7)

A final classification layer encompassing a sigmoid activation function and dropout is utilized to produce the probability of diabetes occurrences. The model has been trained by applying Binary Cross-Entropy (BCE) loss described as:

$L=-[y \log (\hat{y})+(1-y) \log (1-\hat{y})]$  (8)

Now, the parameter $\hat{y}$ points to the predicted probability, and $y$ indicates the true class label. This architecture effectively performs feature interactions and non-linear patterns of decision-making with the help of embedding FT-Transformer and MLP as classification layers, thus improving the accuracy of timely diabetic diagnosis by employing features obtained from an IoT sensor network. Figure 2 represents the architecture of the FT-Transformer with an MLP classification head.

Figure 2. Structure of the Feature Tokenizer Transformer (FT-Transformer)-Multi-Layer Perceptron (MLP) classification model

3.5 AdamW optimization

This section describes the AdamW optimization method used to improve model training and enhance generalization performance. AdamW is an enhanced form of the Adam optimizer that improves model training by avoiding overfitting and increasing generalization [22]. It uses L2 regularization instead of standard Adam, decouples weight decay and gradient update with L2 regularization so as to update weights more effectively. Weight decay can be effectively applied in the FT-Transformer early diabetes prediction model, which guarantees consistent convergence and better performance in classification accuracy. The weight update rule is derived from Eq. (9):

$\theta_t=\theta_{t-1}-\eta\left(\frac{m_t}{\sqrt{v_t}+\epsilon}+\lambda \theta_{t-1}\right)$ (9)

The AdamW optimizer improves training efficiency and model generalization by effectively handling weight decay, resulting in enhanced classification performance for early diabetes prediction.