Assessing the Impact of Data Source Heterogeneity on Machine Learning-Based Stroke Risk Prediction: A Multi-Dataset Generalization Study

This study investigated the impact of data source heterogeneity on machine learning-based stroke risk prediction using a rigorous, multi-step methodology. The approach involves the selection of three distinct public health datasets, followed by a standardized data preprocessing pipeline, comparative analysis using Principal Component Analysis (PCA), and a comprehensive evaluation of multiple machine learning models.

2.1 Datasets

Three open-source publicly available datasets were used for this comparative analysis [18-20]. All datasets were sourced from open-access platforms and contained patient health information relevant to stroke risk as follows:

Dataset 1 (Kaggle Stroke Prediction Dataset): It contains 5,110 patient records with 12 features, including demographic information, lifestyle factors, and specific clinical measurements such as avg, glucose level and body mass index (BMI). The target variable, stroke, indicates a stroke event. This dataset is sourced for educational purposes and was used for this study for comparative purposes to demonstrate the purpose of the investigative approach used. The data was recently modified to be used for educational and research purposes [20].

Dataset 2 (Mendeley Stroke Prediction Dataset): It includes a total of 4603 subjects who fulfilled the inclusion and exclusion criteria from NHANES. Of these, 362 individuals (7.86%) were diagnosed as stroke patients, while 4,241 individuals (92.14%) were identified as non-stroke patients, offering a much more detailed set of clinical and lifestyle variables, including lab results like fasting glucose, total cholesterol, and various nutrient intake metrics [19].

Dataset 3 (Framingham Heart Study Dataset): It includes 4,240 records,16 columns, and 15 attributes. While originally designed to predict the 10-year risk of coronary heart disease (CHD), the target variable, TenYearCHD, was renamed to stroke to ensure consistency in classification tasks across all study datasets. It comprises records from the long-term Framingham Heart Study, which focuses on cardiovascular disease. Key features include age, systolic blood pressure (BP), diastolic BP, and glucose. The target variable, TenYearCHD, was renamed to stroke to maintain consistency across the datasets [18].

2.2 Data preprocessing and alignment

As the datasets were from different sources with different categories, a standardized preprocessing pipeline was applied to each dataset to ensure a fair and consistent comparison. As a first step, a dataset summary table (Table 1) was generated to document the number of samples, features, and the percentage of stroke cases in each dataset. For each dataset, categorical features were encoded using LabelEncoder, and missing numerical values were assigned the mean of their respective columns. Model training and evaluation were conducted using the full, unique feature set of each respective dataset. A core set of features (age, gender, hypertension) was identified for initial alignment reference and statistical analysis. Furthermore, the full range of features was utilized by the machine learning models to capture a comprehensive, medically accurate representation of predictive drivers.

Table 1. Summary of dataset characteristics and statistical significance of risk factors

This ensured that all models had a common, medically relevant starting point in spite of their differences (like Dataset 2 having 36 features and Dataset 1 having 12). However, the complete feature set from each source was preserved for the subsequent feature importance analysis, allowing for a more clinically insightful interpretation.

Although basic risk factors are present, the performance comparison was fair. Finally, each prepared dataset was partitioned into feature matrices (X) and target vectors (y) to facilitate model training.

To address the inherent heterogeneity across the selected datasets, a structural alignment of clinical features was performed. As detailed in Table 2, while several core physiological markers, specifically age, hypertension, heart_disease, glucose level, and BMI, were present across all three cohorts, other clinical and lifestyle variables were unique to specific data sources. This feature mapping served as the foundation for our standardized preprocessing pipeline. To maintain structural consistency, missing features in specific cohorts were addressed through a systematic imputation approach. Recent applications of cross-dataset validation demonstrate that these approaches are critical for confirming the clinical applicability of binary classification models [21-23]. This alignment ensures that our comparative analyses were conducted on a consistent set of medically relevant risk factors while preserving the unique predictive information present in each dataset. Furthermore, to address the impact of class imbalance on model predictive performance, the SMOTE was directly integrated within our cross-validation pipeline. By employing an ImbPipeline, synthetic samples were generated only during the training phase, preventing data leakage into the validation folds (Table 3). While the Framingham Heart Study is primarily recognized for cardiovascular disease prediction, the TenYearCHD (10-year risk of CHD) variable was utilized as a proxy for stroke risk in our comparative analysis.

Table 2. Feature alignment across study datasets

Table 3. Hyperparameter search space and optimization settings

However, given the strong correlation between shared cardiovascular risk pathways, this proxy allows for a rigorous assessment of model generalizability across heterogeneous patient cohorts [21-23]. This decision is supported by the significant pathophysiological overlap between CHD and ischemic stroke; both conditions share the same core atherosclerotic risk factors, including hypertension, age, diabetes, and hyperlipidemia.

2.3 Machine learning workflow

ML process began with data ingestion and preprocessing, followed by exploratory analysis using PCA. Models were then trained and evaluated using a cross-validation and hyperparameter tuning pipeline, and the best-performing models were used for performance metric calculation and feature importance analysis. The overall workflow for the machine learning analysis is summarized in Figure 1.

1.png

Figure 1. Machine learning (ML) workflow algorithm for dataset analysis

2.4 Principal Component Analysis

To visualize data distribution, PCA was applied to each dataset to identify potential heterogeneity, which is one of the main contributions of this study. StandardScaler was used as a feature matrix standardization tool to ensure equal contribution to the analysis [24-26]. A two-component PCA model was subsequently fitted to this standardized data. The resultant principal components were projected onto a two-dimensional scatter plot. This plot enabled a visual inspection of the clustering patterns and the degree of separation between stroke and non-stroke cases. The explained variance ratio for each principal component was also recorded to quantify the informational value of this visualization.

2.5 Model selection and evaluation

To predict stroke risk, Logistic Regression, RF, K-Nearest Neighbors (KNN), and SVM were employed as robust classifiers. For each model, an end-to-end pipeline was constructed to ensure methodological rigor. To address the significant class imbalance observed across the cohorts, particularly the low prevalence of stroke cases in the Mendeley dataset, SMOTE was integrated. Crucially, to prevent data leakage, SMOTE was applied exclusively to the training subset within each fold of our cross-validation process.

The models were evaluated using 5-fold cross-validation with a GridSearchCV approach to optimize hyperparameters (detailed in Section 3.2.2). Strict pipeline nesting was maintained: StandardScaler, the SMOTE resampling step, and the GridSearchCV hyperparameter tuning were executed solely on the training data of each respective fold. This ensured that the feature scaling, synthetic sample generation, and parameter selection were independent of the test set, providing an unbiased estimate of the model’s true clinical performance.

2.6 Performance metrics and feature importance

The performance of each model was evaluated using a comprehensive suite of metrics designed to provide a multi-dimensional assessment of diagnostic capability:

2.6.1 Accuracy

This metric measures the proportion of correctly classified instances. However, it can be misleading in imbalanced datasets as the model may prioritize the majority class ("No Stroke") to artificially inflate the score.

2.6.2 Precision

Representing the ratio of true positives to total positive predictions, precision is vital for clinical resource management. High precision minimizes unnecessary follow-up interventions and reduces patient anxiety associated with false-positive results.

2.6.3 Recall (Sensitivity)

This metric calculates the ratio of true positives to total actual positive cases. Maximizing recall is critical in stroke prediction to ensure that potential stroke events are not overlooked.

2.6.4 F1-score

Serving as the harmonic mean of precision and recall, the F1-score provides a balanced metric. It penalizes models that exhibit poor performance in either precision or recall, making it a robust measure for evaluating binary classification models.

2.6.5 Area Under the Curve-Receiver Operating Characteristic

The Area Under the Curve-Receiver Operating Characteristic (AUC-ROC) quantifies the model’s inherent ability to discriminate between stroke and non-stroke classes. By assessing performance across all classification thresholds, AUC-ROC remains a key indicator of model strength despite inherent class imbalances.

Feature importance analysis using the RF model enhanced transparency and clinical interpretability, which is the top-performing classifier across all datasets. This analysis categorizes clinical variables, such as age, glucose levels, and hypertension, by their contribution to the model's predictive output. The importance was visualized via bar charts to allow for a direct comparison of the most influential clinical factors across the three distinct cohorts. Given the variation in feature sets across the Kaggle, Mendeley, and Framingham datasets, utilizing the full feature set ensures that findings remain medically relevant to each specific cohort. Identifying these key clinical drivers is essential for the eventual clinical adoption of machine learning tools; our results consistently highlight age and glucose levels as the primary determinants of stroke risk.

This aligns with modern additive explanation frameworks [27], which emphasize the need to move beyond 'black-box' predictions toward transparent clinical decision support.

The primary objective of this study was to evaluate the role of data heterogeneity in machine learning-based stroke risk prediction. Our results demonstrate that data from diverse sources, while addressing the same clinical problem, present unique statistical distributions and feature correlations with profound implications for predictive modeling.

Statistical analysis confirmed that well-established clinical risk factors, specifically age, hypertension, heart disease, and elevated glucose levels, are significantly associated with stroke across all three cohorts (Table 2 and Table 3). These findings, consistent with medical literature, provide a robust foundation for our machine learning analysis. The significant p-values and logistic regression ORs (Table 4) provide quantitative evidence of the magnitude of these relationships, confirming that as patients age or develop hypertension, stroke risk increases. Furthermore, while the use of synthetic datasets (e.g., Mendeley cohort) offers a platform for testing model robustness [28], our results underscore the necessity of validating these models against real-world benchmarks to ensure clinical utility [29].

4.1 Interpretation of clinical risk factors

Our machine learning framework, specifically the RF importance analysis, consistently identified age, blood glucose levels, and hypertension as the primary predictors of stroke risk across all cohorts. This result is strongly validated by our independent statistical evaluations; both the t-tests (for continuous variables like age and glucose) and Chi-Square tests (for binary indicators like hypertension) confirmed statistically significant differences (p < 0.05) between stroke and non-stroke populations. The convergence of these two distinct methodologies, machine learning feature ranking and univariate statistical testing (Tables 2–4), enhances the clinical reliability of our findings. Biologically, these features represent key facets of cardiovascular health: age reflects the cumulative effect of arterial degradation, while hypertension and glucose levels are direct drivers of vascular endothelial dysfunction and plaque instability.

4.2 Performance analysis and computational practicality

While our RF model performed well, simpler models like SVM and Logistic Regression initially struggled with predictive sensitivity. This is likely due to the severe class imbalance inherent in stroke datasets, a challenge well-documented in the literature. To address this, the impact of synthetic oversampling techniques was explained, specifically SMOTE [30-32]. Our results demonstrate that this integration significantly enhanced the sensitivity of the linear models, confirming that mitigating class skewness is a prerequisite for achieving reliable classification in heterogeneous clinical datasets.

In evaluating the practical utility of our proposed models, the computational requirements of our pipeline were assessed. As expected, non-linear ensemble methods, specifically RF, demonstrated higher training complexity and execution time compared to linear classifiers. This demonstrates that a single, powerful model can effectively generalize across heterogeneous data sources by dynamically weighting the most relevant features available [30]. However, this increased computational cost is marginal in the context of stroke risk prediction, where the priority is maximizing sensitivity to identify minority-class stroke cases. The trade-off favoring the superior F1-score and Recall of RF over the computational efficiency of Logistic Regression is both necessary and justified for clinical efficacy. This was justified given that these models are intended for clinical support and not high-frequency real-time processing.

4.3 Study limitations

While our study provides a framework for addressing data heterogeneity, it is subject to certain limitations. As detailed in the Methodology (Section 2.2), our use of the Framingham Ten-Year CHD variable as a proxy for stroke risk represents a surrogate endpoint. While clinically justified by the shared pathophysiology of cardiovascular and cerebrovascular diseases, future research should prioritize validation using direct stroke diagnostic codes to further refine predictive accuracy across diverse clinical datasets. Furthermore, future work should focus on developing models that are not only accurate but also inherently interpretable (e.g., using SHAP values), ensuring that predictions can be readily translated into actionable clinical decisions.