An Advanced AI Framework for Mental Health Diagnostics Using Bidirectional Encoder Representations from Transformers with Gated Recurrent Units and Convolutional Neural Networks

An application of deep learning for mental health diagnostics has shown promise in recent years as a way to transform conventional methods of assessment and therapy. Mental health issues are a major global burden that impairs people's quality of life and puts pressure on global healthcare systems [1]. Even with the severity and frequency of these illnesses, prompt and precise diagnosis is still difficult to come by, which frequently causes delays in starting therapy and worse-than-ideal results [2]. Image identification, natural language processing, and medical diagnostics are just a few of the fields in which deep learning, a subset of artificial intelligence that draws inspiration from the structure and functions of the human brain, has shown impressive results [3]. Leveraging large datasets and complex neural networks, deep learning algorithms have shown the potential to enhance the accuracy and efficiency of mental health assessments [4].

The capacity of machine and deep learning algorithms to identify patterns is one of its key advantages. These techniques show some promise in identifying generally applicable trends among those who suffer from mental health disorders. For example, Carrillo et al. [5] demonstrated how transcribed textual data may be used to accurately identify healthy controls from patients experiencing depression using a Gaussian Naive Bayes classifier with an F1-score of 0.82. Considering the known challenges in identifying mental health issues, psychiatrists can gain from depression diagnosis methods. Mental health disorders lack objective illness indicators, in contrast to other areas of medicine [6]. One of the main difficulties in diagnosing psychopathology is the absence of an objective marker [7]. Because populations with the same diagnosis sometimes have significantly different symptoms, current diagnostic techniques are under scrutiny [8].

Unsupervised learning techniques assist in the identification of several subgroups of depression or maybe new diagnoses. Drysdale et al. [6] examined functional connectivity across patients diagnosed with depression to evaluate depressive heterogeneity using hierarchical clustering, an unsupervised learning technique. Unsupervised techniques allow researchers to discover previously unknown links, even if supervised techniques are used in the majority of the research examined in this study. They distinguished four distinct depression biotypes using fMRI scans. It has been shown that the responses to rTMS therapy vary throughout these biotypes. Every subtype probably represents a distinct illness because they all responded to therapies in various ways. The potential of artificial intelligence systems to facilitate the shift to new diagnostic taxonomies is demonstrated by this work.

Contemporary computing techniques not only facilitate the identification and diagnosis of mental health disorders but also present an opportunity to customize therapy recommendations. To determine which antidepressant is best for a patient, physicians now use a trial-and-error method [9, 10]. Chang et al.'s revolutionary research [11] shows that psychiatrists can assess an antidepressant drug's expected effect before administering it. Their research demonstrates that the Antidepressant Response Prediction Network, or ARPNet, is an artificial neural network that can accurately forecast an antidepressant's side effects before starting therapy. The potential for patient-specific treatment is increased by these technologies. Artificial intelligence's initial goal was to mimic human functions artificially [12]. Such research aimed to develop symbolic artificial intelligence in its early stages. "Carry out a series of logic-like reasoning steps over language-like representations" was the stated objective of work on symbolic artificial intelligence [13].

That being said, most artificial intelligence researchers are no longer primarily interested in symbolic artificial intelligence. Instead, pattern recognition based on artificial neural networks is currently dominating the field. A significant amount of the current research on neural networks is based on the first example of a perceptron, which can be found in Rosenblatt's seminal work [14]. The emergence of deep learning can be attributed to the expansion of these networks due to technical breakthroughs [15]. In deep learning, "depth" refers to an artificial neural network's quantity of hidden layers. However, there is disagreement on what constitutes a "deep" neural network [16]. Sheu [17] has proposed a deep neural network must include at least three layers: an input layer, a hidden layer, and an output layer. However, before classifying a network as a deep neural network, modern researchers typically need to identify multiple hidden layers.

By tackling these goals, this study hopes to add to the expanding corpus of research on the correlation between advanced technologies and mental health that offer insights into the deep learning technology may revolutionize mental health diagnoses. This paper explores the potential of deep learning technology to revolutionize mental health diagnostics by providing a comprehensive review of existing literature, methodologies, and applications. By analyzing the strengths and limitations of current approaches, we aim to make use of opportunities for future research and development in this rapidly evolving field.

The primary research contributions of this paper are given as follows:

• Examine the state of mental health diagnosis today and the difficulties in using conventional methods of assessment.
• Design the BERT- GRU-CNN framework to identify the patterns in the dataset to diagnose mental health issues.
• Design a hybrid framework BERT-GRU-CNN with modified layers with clinical data settings for a range of mental health conditions, such as bipolar disorder, schizophrenia, anxiety, and depression.
• Evaluate the designed model for the applicability, benefits, drawbacks, and moral issues surrounding the use of deep learning technology in mental health diagnosis.

3.1 Data gathering

Data gathering is the initial stage of the proposed model design and it is necessary to gather the data required to design and evaluate the model. The gathered post data from six conditions relating to mental health are depression, anxiety, bipolar, BPD, schizophrenia, and autism, each of which is said to be linked to a particular disorder. To further evaluate the proposed framework with general health information, also gathered post data from the most popular Subreddit for health-related content. We gathered the user IDs from every Subreddit where at least one post had something to do with mental health. Using the web scrapping tool, we gathered titles and posts in addition to user IDs. In total, 348,637 people who posted 732,384 messages in the different Subreddit provided data for the current study.

3.2 Procedure for pre-processing data

Data pre-processing is the second very important step in the research for cleaning the post-data collection from social media. Following data collection, every title and related post was merged. For every post, we eliminated extraneous punctuation and white space. Then, we filtered frequently used words (stop words) and tokenized user postings using the Python implementation of the Natural Language Toolkit.

We made advantage of the NLTK library's stop word list. Common terms like pronouns, prepositions, and conjunctions that are usually eliminated in natural language processing tasks are included in this list. As a starting point, we modified the typical stop word list by eliminating words like "anxiety" and "depression", which are pertinent to discussions about mental health, and adding filler words like "lol", "umm", and "lol" that are commonly used in casual writing.

3.3 BERT-based GRU-CNN model

The proposed model is a deep learning and natural language processing model for predicting mental health disorders. The model is designed by using three models BERT, GRU, and CNN. The complete design is explained in the below section. The initial stage of the model is using the BERT natural language model to process the data by training in both directions. The Bidirectional Encoder Representations from Transformers (BERT) deep learning model is intended to jointly condition on both left and right contexts in every layer in order to pre-train bidirectional representations. Natural language processing (NLP) activities including named entity identification, sentiment analysis, and question answering have been transformed by BERT, a Google creation. The essential component of BERT's design is the self-attention mechanism, which enables the model to assess each token's significance in relation to every other token in a sequence. The dot product of a token's T vector and the V vectors of every other token in the sequence is used to calculate the attention scores. To stabilize gradients, the scores are scaled using the square root of dimensionality of K. An attention score AS is calculated using the Eq. (1):

$A S=\frac{T V^K}{\sqrt{d_v}}$        （1）

The input to the BERT is a sequence of tokens, that are the combination of the words and special tokens. Each token i is then converted into three embedding token embedding, segment embedding, and position embedding as shown in Eq. (2):

$E i=T i+S i+P i$          （2）

The position-wise feed-forward network and multi-head self-attention are the two sublayers that make up each encoder layer. Between each of the two sublayers, the normalization and residuals layers are applied. In this study, we employ the BERTBASE model, which has 110 million parameters, 12 self-attention heads, a hidden size of 768, and an encoder transformer blocks of twelve. We employ the BERT model's feature-based methodology to get a text data in quantitative representation. This is a feature-based approach hence takes a pre-trained model and uses it to extract fixed features.

The feature-based technique offers two benefits over the fine-tuning approach. First, a feature-based approach may be used to add several unique model architectures to each job. This is done because not all issues can be promptly resolved by the transformer encoder architecture. Second, because the precomputation representation procedure is only done once, the feature-based method increases computational efficiency.

The variant of RNN considered as a simplest form of the conventional LSTM is the gated Recurrent Unit. The GRU introduces a gate called the reset gate, that helps combine the input. It also merges the functionality of the forget and input gates from the LSTM into a single gate, known as the update gate. Additionally, GRU creates a single hidden state by fusing two LSTM states the hidden state and the cell state. At the $t$ th time step, GRU, like LSTM, feed input as a vector $x_{\mathrm{k}}$ and a hidden state $h_{(t-1)}$. Finding out what data from the prior time step has to be replicated is the first stage in the GRU process. The update gate $z_t$ as indicated in Eq. (3), is used for this phase.

$z_t=\sigma\left(W_{z x} x_t+W_{z h} h_{t-1}+b_z\right)$          (3)

That data from the previous time period should be discarded is decided in the following step, which is to reset the gate $r_t$. The Eq. (3) illustrates this phase. The hidden state candidate value $h_t$ is then determined using the reset gate's findings, which aids in preserving important historical data. The candidate value of concealed state is calculated using Eq. (4).

$r_t=\sigma\left(W_{r x} x_t+W_{r h} h_{t-1}+b_r\right)$         （4）

$\tilde{h}_t=tanh\ tanh \left(W_{h x} x_t+r_t \odot W_h h_{t-1}+b_h\right)$            (5)

Finding the hidden state $h_t$, that is required to collect and forward data to the following stage is the final step. The hidden state may be computed using the formula in Eq. (5):

$h_t=z_t \odot h_t-1+\left(1-z_t\right) \odot \widetilde{h}_t$            (6)

Here, $W_{\{z z, z h, r r, r h, h x, h\}}$ considered as a weighted matrix and $b_{\{z, r, h\}}$ considered as a biased vector [6].

The GRU processes sequential data and outputs a hidden state for each time step. This hidden state is a vector representation that captures temporal dependencies and sequential patterns in the data. The GRU's output can be a sequence of hidden states: Each time step has a corresponding output. Or a final hidden state: Summarizes the entire sequence. It will be feed as input to the CNN.

An outline of the CNN-based approach that has been suggested. This model's convolutional layer receives word vector input and applies a variety of filters with five different filter sizes. To avoid over-fitting problems, we also used a dropout rate of 0.25. Similar to many f filters, U will use the convolution layer to evaluate the input and determine the salient features of each region within a given region size. Assume that the $j$ th word vector representation is represented by $y_j$ and that the vector combination of the words $y_j$ to $y_{j+k-1}$ is represented by $Y_{[j: j+k-1]}$. The feature vector $V=$ $\left[v_1, v_2, \ldots \ldots \ldots, v_{m-k+1}\right]$ for each filter is determined using Eq. (2), where l and m stand for the matrix's rows and columns and n is a nonlinear activation function. The convolutional layer sends the output to max pooling layer as shown in the form of Eq. (7).

$v_j=n\left(\sum_{m=1}^k \sum_{l=1}^a Y_{[j: j+k-1] m, l} . U_{m, l}\right)$           (7)

The max-pooling layer, which has a size of 128 and uses the maximum values inside the CNN filters. The output of the convolutional layer is processed by the max pooling layer by extracting the most prominent features v , thus $\hat{v}=\max \{\mathrm{v}\}$.

GRU-CNN was one of the hybrid deep learning models employed in this study. Prior to doing sentiment analysis with these models, BERT is used to convert the input text data into a numerical representation. GRU-CNN based on BERT. Every input sentence in GRU-CNN models passes the procedure. To identify the key characteristics in the data, GRU first processes the text representation that is the result of the BERT model. After that an output is fed into the CNN model that considers the data order to produce a new representation. The GRU-CNN models' architecture is made to handle mental health data with a variety of significant feature patterns. Every input sentence goes through a GRU-CNN model procedure. Here, GRU learns the features by observing the sequence of features quality in the data. CNN then receives the GRU output and searches the data for key characteristics. Additionally, the output is mapped into two categorization classes using a fully linked layer.