A Multitask Neural Network Framework for Multistage Alzheimer's Disease Diagnosis and Hippocampal Segmentation Based on Neuroimaging Modalities

AD is one of the causes of dementia, which is a brain disorder caused by damage to nerve cells in the brain, leading to mental decline and memory loss. AD is a progressive disease, as the nerve cells are damaged, the patient gradually experiences changes in memory, language, mood, personality or behavior, eventually affecting basic bodily functions such as walking and swallowing, and can be life-threatening [1, 2]. Studies have shown that the average survival time after diagnosis is 4-8 years [3]. The onset of AD can be divided into three stages: NC, MCI due to AD, and dementia due to AD. Patients with MCI often exhibit mild brain lesions, but do not show significant symptoms. Approximately one-third of MCI patients will progress to AD within five years. There is no effective way or drug to treat AD, but some studies have shown that some patients with MCI do not have additional cognitive decline or revert to normal cognition [4]. Preventive treatment at the MCI stage can be effective in slowing or stopping the AD progress. Thus, a multistage diagnosis of AD can assist in differentiating these three groups, which is helpful in determining appropriate therapeutic interventions.

Neuroimaging data can be employed as diagnostic data in computer-aided diagnostic techniques for AD. For instance, AD-related neuronal loss results in anatomical alterations in the patient's brain tissue that can be identified by structural MRI scans [5, 6]. For this reason, MRI is a frequently used image data set. Fluorodeoxyglucose positron emission tomography (FDG-PET) is another medical imaging modality that can identify abnormalities in glucose metabolism caused by AD. As a result, the distinctive pattern of electron glucose metabolism in PET images can be used to distinguish between patients and NC [7, 8]. Furthermore, aberrant levels of beta-amyloid and tau in CSF are established biomarkers of AD, making beta-amyloid and tau buildup useful information for the diagnosis of AD [9]. While these medical data can independently diagnose AD, integrating multimodal data significantly enhances diagnostic precision [10, 11].

Currently, the binary diagnosis of AD has achieved a high accuracy rate, however, the multistage diagnosis task, which can distinguish AD, MCI and NC simultaneously, has more practical value in clinical applications. But the accuracy of computer-aided multistage diagnosis of AD is still low [12-14] due to the slight variations in the pathogenesis of AD between adjacent stages, particularly in the early stages of MCI where the pathological changes are very small in comparison to normal individuals. In order to more precisely distinguish the different stages of AD, multitask deep learning models have been used to improve diagnosis accuracy [15, 16], i.e., to acquire the diagnostic results while also outputting additional metrics, such as mini-mental state examination (MMSE) and clinical dementia rating (CDR). Given that AD results in hippocampal atrophy, and the hippocampal regions are considered to be the main landmark areas for lesions in Alzheimer's studies [17, 18], allowing the neural network model to learn the structural features of the hippocampus will improve the ability to diagnose AD of different stage. This paper proposes a novel multitask deep neural network model named Classification and Segmentation Multitask Network (CSMT-Net), which adds a segmentation task to segment the hippocampal structure along with the MMSE, CDR regression and AD classification tasks. The experiments of multistage diagnosis (AD vs. MCI vs. CN) were conducted. Based on the results of the experiments, the accuracy of AD multistage diagnosis can be further enhanced by including the hippocampus segmentation task.

The main highlights of this paper are as follows:

1. This work proposed a deep learning-based approach for AD multistage diagnosis. A multitask deep neural network named CSMT-Net was designed for AD-related feature extraction with neuroimaging, such as MRI and FDG-PET.
2. Hippocampus segmentation was utilized as one task of CSMT-Net for structural MRI, so that the network could learn about the AD-related structural lesion information, which could indicate the progress of AD and is useful for AD multistage diagnosis.
3. A machine learning framework consisting of PCA and ELM was developed for AD multistage diagnosis using multimodal data, including the neuroimaging features produced by CSMT-Net and additional clinical data, such as CSF biomarkers, genetic factors, and demographic information.

In this section, a multimodal framework for AD multistage diagnosis is introduced, Figure 1 displays the architecture of this framework. The framework comprises two components: the CSMT-Net part and a machine learning part. The CSMT-Net, which is pre-trained with classification, regression and segmentation tasks, extracts deep features from preprocessed 3D MRI and 3D PET images. Subsequently, the deep features of MRI and PET are reduced to 5 features each using the PCA algorithm. The neuroimaging deep features are then concatenated with hippocampus volume, CSF biomarkers, apolipoprotein E4 (APOE4), a genetic risk factor for AD [32], and demographic data including age, gender, and education, before an ELM classifier is employed with these features to generate a multistage diagnosis.

Figure 1. The overall framework of the CSMT-Net based multimodal AD multistage diagnosis

3.1 Neuroimaging preprocessing

The images in this paper were acquired from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (https://ida.loni.usc.edu/). Due to variations in size, head position, and orientation of the MRI and PET images, preprocessing of these neuroimages was required. Preprocessing involves skull stripping and rigid registration. The MRI images underwent N4 bias field correction with the ANTs tool, skull stripping with FSL software, rigid registration to the MNI152 template using the IRTK tool, and cropping to produce a 128×160×128-sized 3D brain image. The PET images were initially registered with the raw MRI image, skull stripped using the MRI skull stripping mask, aligned with the registered MRI, and then cropped to a size of 128×160×128. Data augmentation techniques were utilized during the neural network training phase to expand the training dataset. This involved randomly adjusting the cropping center point, flipping images along the left/right axis, and applying random rotations within a range of ±20 degrees in three dimensions.

3.2 The architecture of the CSMT-Net

The architecture of CSMT-Net presented in this paper is illustrated in Figure 2. It begins with an input layer that processes the input image. This layer includes a 3D convolutional layer with a convolutional kernel size of 3×3×3, followed by a batch normalization (BN) layer and a Rectified Linear Unit (ReLU) activation layer. Subsequently, maximum pooling is applied, which decreases the feature map size to 64×80×64 and generates 16 feature maps. Following the input layer, there are four 3D convolution blocks that utilize a residual structure. The 3D convolution block consists of three sets of convolutions, BN layer, and ReLU layer. The input is combined with the output of these convolutions through residual linking. The combined feature maps are then sent through another set of convolutions, BN layer, ReLU layer, and a maximum pooling layer. The feature map size is reduced by half in each 3D convolution block, while the number of feature maps is doubled. As a result, the fourth residual convolution block produces 256 feature maps with dimensions of 4×5×4. A global average pooling operation is performed to calculate the average values of all feature maps, resulting in a 1×256 vector. A 256×512 fully connected layer is followed by a dropout layer, a ReLU activation layer, and another 512×32 fully connected layer to produce a 1×32 vector, which represents the deep features of the neuroimage. During the network training phase, this feature vector is utilized to create training targets for the multitask learning, including MMSE, CDR, and classification label, through two 32×1 fully connected layers and one 32×3 fully connected layer.

Figure 2. The architecture of the CSMT-Net

A U-Net-similar up-sampling branch is added to the backbone network for the purpose of segmenting the right and left hippocampus, for the network designed for MRI feature extraction, as depicted in Figure 2. The up-sampling module takes the input, up-samples it using an inverse convolution layer, and combines it with the relevant feature maps from the backbone network using skip links. The resulting output is passed through two sets of convolutions, a BN layer, and a ReLU activation layer. The up-sampling module produces feature maps that are twice the size of the input. After four up-sampling modules, 16 feature maps of size 64×80×64 are generated. These feature maps are further up-sampled by the final inverse convolution, resulting in segmentation outputs of size 128×160×128. The hippocampus segmentation results will be utilized as one of the tasks in the training of the MRI network. As manual segmentation of hippocampal labels is difficult to achieve, the segmentation tool in FSL software was utilized to segment the hippocampus from the MRI, and the segmentation results are then used as the training labels for the segmentation task. While training the MRI network for segmentation, the network can acquire segmentation skills and morphological knowledge similar to the FSL software, enhancing its capacity to extract valuable information from MRI data. The hippocampal segmentation output is also utilized to calculate the hippocampal volume, which can serve as an AD biomarker.

3.3 Loss function

When training the CSMT-Net, the loss function of the multitask training objective must be combined. The PET network has three tasks: multi-classification, regression of MMSE and CDR. A fourth task, which is segmenting the hippocampus, is included in the MRI network. The loss functions for these tasks are built as follows.

The multi-classification task utilized the multi-margin loss, which can enlarge distances of interclass and reduce intra-class variations simultaneously [33]. The loss function of multi-margin loss can be expressed as:

${{L}_{c}}=\frac{1}{C}\sum\limits_{i=0\And i\ne {{y}_{n}}}^{C-1}{\max {{\left( \begin{align}  & 0,w[y]*(\text{margin} \\ & -x[y]+x[i]) \\\end{align} \right)}^{p}}}$     (1)

where, C denotes the group number, specifically set to 3. x[y] represents the output of the correct group, and x[i] are the outputs of other groups. For the situation of sample imbalance, w[y] is the weight for each group, which is set to 1. p and margin are set to the default value, which is 1.

For the tasks of MMSE and CDR regression, the loss function is squared loss, which can be expressed as:

$\begin{matrix}   {{L}_{mmse}}={{\left( {{y}_{mmse}}-{{{\hat{y}}}_{mmse}} \right)}^{2}}, & {{L}_{cdr}}={{\left( {{y}_{cdr}}-{{{\hat{y}}}_{cdr}} \right)}^{2}}  \\\end{matrix}$     (2)

where, y represents the output value, and$\hat{y}$represents the true value.

For the task of hippocampus segmentation, the binary cross-entropy loss for each voxel was calculated and averaged as the segmentation loss function, which can be expressed as:

${{L}_{seg}}=-\frac{1}{N}\sum\limits_{i=1}^{N}{\begin{align}  & {{y}_{i}}\log \left( p({{y}_{i}}) \right) \\ & +(1-{{y}_{i}})\log \left( 1-p({{y}_{i}}) \right) \\\end{align}}$     (3)

where, N represents the number of voxels, y represents the ground truth label, and $p\left(y_i\right)$ represents the output probability of label.

Finally, we combine the above loss functions by weighting to get the final loss function as follows:

$Loss=\frac{1}{4}\left( \begin{align}  & {{w}_{c}}\cdot {{L}_{\text{c}}}+{{w}_{mmse}}\cdot {{L}_{mmse}} \\ & +{{w}_{cdr}}\cdot {{L}_{cdr}}+{{w}_{seg}}\cdot {{L}_{seg}} \\\end{align} \right)$     (4)

where, w_c, w_mmse, w_cdr, w_seg are the weighting coefficients of the four loss functions. Take note that the different loss functions often operate on vastly different scales, the individual loss functions are scaled by these weighting coefficients to bring them into a comparable magnitude. This ensures that each task's contribution to the overall multitask loss is more balanced, preventing one task from overshadowing others purely due to its inherent scale. According to the output scale of each loss function, w_c was set to 1, w_mmse was set to 0.03, w_cdr was set to 0.1, w_seg was set to 1.

3.4 Classifier

The proposed method involves extracting MRI and PET features using a well-trained CSMT-Net that outputs 32 deep features. The features are initially processed using PCA to merge the highly correlated features [34]. Five major components are kept, while the rest are removed as noise. So 5 deep features are extracted from each MRI and PET image. These neuroimaging features are combined with additional biomarkers and demographic data, then inputted into a classifier for AD diagnosis. A classifier using ELM with Gaussian kernels [35] is created to do multistage classification of AD using multimodal data. The ELM algorithm can be described as follows: assuming there are N training samples [x₁, x₂, ⋯, x_N]_, in which, x_n represents the n-th sample consisting of M features. Y∈R^N×G is a one hot ground truth label matrix for N samples of G classes. Upon receiving a new sample x, the label of x can be predicted as

$f(x)=\left[ \begin{matrix}   \text{K}\left( \mathbf{x},{{\mathbf{x}}_{1}} \right)  \\   \text{K}\left( \mathbf{x},{{\mathbf{x}}_{2}} \right)  \\   \vdots   \\   \text{K}\left( \mathbf{x},{{\mathbf{x}}_{\text{N}}} \right)  \\\end{matrix} \right]{{\left( \mathbf{\Omega }+\mathbf{I}\text{/}C \right)}^{-1}}\mathbf{Y}$     (5)

where, the variable C is a regularization coefficient set to 1. The variable γ is a parameter of the Gaussian kernel, which was set to 10 times of M in this study, and K(x, x_n) is the Gaussian kernel described as:

$\operatorname{K}\left( \mathbf{u},\mathbf{v} \right)=\exp \left( -{{\left\| \mathbf{u}-\mathbf{v} \right\|}^{2}}\text{/}\gamma  \right)$     (6)

and Ω is an N×N kernel matrix that is calculated with N training samples:

$\mathbf{\Omega }=\left[ \begin{matrix}   \begin{matrix}   \operatorname{K}\left( {{\mathbf{x}}_{1}},{{\mathbf{x}}_{1}} \right)  \\   \operatorname{K}\left( {{\mathbf{x}}_{2}},{{\mathbf{x}}_{1}} \right)  \\   \vdots   \\   \operatorname{K}\left( {{\mathbf{x}}_{\text{N}}},{{\mathbf{x}}_{1}} \right)  \\\end{matrix} & \cdots  & \begin{matrix}   \operatorname{K}\left( {{\mathbf{x}}_{1}},{{\mathbf{x}}_{\text{N}}} \right)  \\   \operatorname{K}\left( {{\mathbf{x}}_{2}},{{\mathbf{x}}_{\text{N}}} \right)  \\   \vdots   \\   \operatorname{K}\left( {{\mathbf{x}}_{\text{N}}},{{\mathbf{x}}_{\text{N}}} \right)  \\\end{matrix}  \\\end{matrix} \right]$     (7)

In this paper, a multitask deep neural network named CSMT-Net is proposed to extract AD-related features from MRI and PET data. These features are combined with CSF biomarkers, Apoe4 genes, age, gender, and education data to enhance the accuracy of AD multistage diagnosis. A U-net up-sampling branch is incorporated into the convolutional neural network backbone framework to perform the hippocampal segmentation task in the MRI feature extraction network. The segmentation task enables the neural network to acquire morphology knowledge about brain tissue structure, enhancing the efficacy of MRI deep features. Furthermore, the volumes of the hippocampus can also serve as AD-related features, enhancing the accuracy of AD diagnosis. Due to the unavailability of manual segmentation for the hippocampus, the FSL tool was used to generate hippocampus segmentation as the segmentation training labels. The FSL segmentation tool, despite not relying on manual segmentation by an expert, is an algorithm with high segmentation accuracy. It incorporates prior knowledge of hippocampal segmentation, enabling the deep neural network to acquire significant MRI structure knowledge and achieve effective segmentation ability.

Multiclass AD diagnosis is a challenging task; the performance of multiclass diagnosis is significantly lower than that of binary diagnosis for AD vs. NC. This is due to the fact that the conversion of NC to MCI, as well as the conversion of MCI to AD, is a gradual process. In this conversion process, there is no obvious boundary between early MCI and NC or late MCI and AD. Therefore, MCI and NC or MCI and AD are easily confused, and it can also be seen from the confusion matrix of the experimental results that most of the classification errors occur between MCI and NC or MCI and AD. Therefore, it is a difficult task to achieve high-precision AD multi-classification. It can also be seen that the key to improving the accuracy of AD multi-classification is to improve the differentiation between MCI and the other two groups, i.e., the classification of NC and MCI and the classification of MCI and AD. For the structure of the hippocampal regions gradually changed during AD progression; by introducing the hippocampus segmentation task, the CSMT-Net could learn information about hippocampal alterations, which could indicate the AD progression and be helpful for the multistage diagnoses of AD.

The experiment results in Table 2 and the confusion matrix in Figure 4 indicate that the classification accuracy of NC and MCI is lower than that of MCI and AD. For this reason, we assume that in the stage of NC and early MCI, the degree of lesion is slight, and the degree of some brain tissue changes or biomarker abnormalities are mild, and these mild changes are not easy to distinguish, so the classification accuracy is lower. Whereas, in the stages of late MCI to AD, there are more lesions, these changes will be more obvious with the increase of the disease, so it is relatively easier to distinguish, so the classification accuracy is higher. However, discriminating MCI patients from the NC cohort is more meaningful because AD patients are easier to treat in the early stage, so a high performance of NC and MCI classification plays a greater role in the prevention of AD.

MMSE and CDR are not utilized as features directly in this paper, because of their substantial correlation with the group labels, as they are primarily employed in the clinical diagnosis of AD. Utilizing them as features can significantly enhance classification accuracy, but it may lead to biased and overestimated findings. Thus, in this paper, the scores were just utilized as training labels for the neural network and not as features.

The proposed deep learning model integrates MRI and PET neuroimaging, CSF features, genetic risk factors, and demographic data for the diagnosis of AD and MCI. Diagnosing MCI is particularly vital as it represents an early stage of AD, enabling timely intervention and preventative strategies. To facilitate the integration of the proposed approach into clinical workflows, the acquisition of multimodal data is most important. Data such as MRI scans, genetic risk factors, and demographic information are generally more accessible and less invasive for patients in routine clinical practice. MRI is a standard neuroimaging technique, and genetic testing and demographic data collection are common procedures. Conversely, obtaining PET neuroimaging and CSF features poses greater challenges. PET scans involve radiation exposure and are more costly, while CSF collection via lumbar puncture is an invasive procedure that carries certain risks and patient discomfort. In a practical clinical workflow, a tiered approach could be adopted. Initial screening might primarily utilize the more accessible data (MRI, genetics, demographics). For cases with inconclusive results or higher suspicion, the more invasive but highly informative PET and CSF data could then be considered to confirm diagnosis or assess disease progression.

Although the proposed approach demonstrates strong predictive performance, it has some limitations. As a deep neural network, the proposed model, particularly when integrating diverse multi-modal inputs, can present challenges in direct interpretability. Understanding the exact contribution and interplay of each specific feature to a given prediction can be complex. The strategies for post-hoc interpretability could be explored in future work to gain deeper insights into the model's decision-making process. While ADNI is a high-quality dataset, real-world clinical data often present greater variability and more extensive missingness. The proposed model's reliance on a comprehensive set of multi-modal data (MRI, PET, CSF, genetic, demographic) means that missing data could impact its applicability outside of well-controlled research settings. Meanwhile, the potential biases inherent in the ADNI cohort, which is predominantly of European descent, limit generalizability to more diverse populations. The proposed model is primarily designed for multi-stage classification based on cross-sectional multi-modal data. While ADNI provides longitudinal data, the proposed approach does not fully leverage the temporal dynamics and progression patterns inherent in these longitudinal measurements to predict disease trajectory or conversion risk over time. The explicit modeling of longitudinal changes could offer a more nuanced understanding of disease progression, and it would be a significant area for our future research and model development.