Augmenting Document Classification Accuracy Through the Integration of Deep Contextual Embeddings

The field of Natural Language Processing (NLP) has garnered substantial attention in recent research, largely due to the advent of deep learning techniques. A central task within NLP is document classification, defined as the process of assigning categories to documents based on their respective contents or characteristics. Traditional text clustering methods, such as bag-of-words models, have historically been prevalent in achieving this task [1, 2]. However, these conventional approaches face limitations. They often struggle to detect sophisticated patterns and interdependencies within textual data, and typically exhibit domain specificity and an inability to handle large data volumes [3].

The constraints of traditional text clustering models become evident in real-world scenarios, such as the processing of customer feedback. Consider a company inundated with diverse customer feedback in the form of product reviews, customer service inquiries, and social media posts. Here, traditional text clustering methods, including bag-of-words models [4, 5], may fail to accurately group similar feedback due to inherent deficiencies in capturing context and meaning.

For instance, when a customer review encapsulates both positive and negative aspects of a product, a bag-of-words model might categorize the feedback based on word frequency rather than capturing the holistic sentiment or meaning. This could potentially lead to misclassification, causing the company to overlook valuable insights. Recent advancements in deep learning have shown potential in surmounting these challenges by utilizing Deep Contextual Embedding Models (DCEMs) [6-9] for text representation. Through the integration of DCEMs, companies can more accurately discern the context and meaning of customer feedback, leading to improved categorization and, ultimately, superior understanding of customer sentiment and needs. These models generate embeddings that capture the contextual information of each word and phrase in a document, thereby enabling clustering algorithms to group similar documents with higher accuracy.

In recent years, deep learning methodologies have demonstrated considerable potential in enhancing the accuracy of natural language processing tasks, including text clustering and document classification [10-12]. Notably, the pre-training of deep bidirectional transformers [6, 8], as epitomized by the BERT model proposed by Devlin et al. [6], has made significant strides in language understanding. Further, Convolutional Neural Networks (CNNs) have been extensively utilized for text categorization, capitalizing on word order to enhance performance [3]. Attention-based models, such as the Transformer model proposed by Vaswani et al. [13], have also demonstrated their efficacy in capturing intricate patterns and relationships within textual data. Despite these advancements, traditional text clustering methods, such as bag-of-words models, retain their prevalence across various domains, although their limitations in capturing contextual information and managing extensive data volumes are well documented [11].

The primary objective of this research is to devise an innovative deep contextual embedding model to address the constraints of existing contextual models and enhance the accuracy of deep text clustering and document classification. The proposed methodology integrates pre-trained contextual models with clustering algorithms, aimed at augmenting the quality of clusters and document classification accuracy. High-level semantic features are extracted from text documents using pre-trained models and subsequently employed as input for clustering algorithms, thus grouping similar documents. This integration improves the capacity to capture the context and semantics of words and phrases, in addition to unravelling the underlying structure and relationships within the text data. Furthermore, it aids in the interpretation and understanding of latent representations in clustering, simultaneously identifying and eliminating noise and outliers, thereby leading to enhanced clustering performance. The novelty of this research paper lies in its integration of pre-trained Deep Contextual Embedding Models (such as GPT-2) with traditional clustering algorithms (e.g., K-Means, DBSCAN), thereby improving text clustering and document classification accuracy and effectively addressing the limitations of conventional methods.

In order to substantiate the proposed methodology, a series of experiments were undertaken utilizing several publicly accessible datasets, including AG News, Reuters-21578, and IMDB reviews. The datasets were selected based on several criteria such as relevance, accessibility, volume, complexity, cleanliness, and their representative nature of real-world data.

The effectiveness of text clustering algorithms and document classification accuracy was assessed using a variety of performance metrics. These included the Silhouette score, Calinski-Harabasz index, Davies-Bouldin index, Homogeneity, Completeness, Normalized mutual information, Accuracy, Precision, Recall, and F1-Score. This assessment was conducted on several labeled datasets with ground truth labels to evaluate the performance of the clustering algorithm. Such experimental evaluation proved pivotal in determining the optimal combination of contextual embedding models for text clustering and document classification accuracy.

The proposed methodology has potential implications across several domains, including data analysis, information extraction, and target recommendation systems. By integrating Deep Contextual Embedding Models (DCEMs) with text clustering algorithms, it is anticipated that the efficiency and accuracy of downstream NLP tasks can be significantly enhanced. In summary, this research represents a significant progression in the field of document classification, highlighting the potential utility of employing DCEMs for a range of NLP applications.

The novelty of our integrated approach lies in combining pre-trained Deep Contextual Embedding Models (e.g., GPT-2) with clustering algorithms (e.g., K-Means, DBSCAN) to effectively captures contextual information, improves clustering performance, and addresses the limitations of traditional methods, resulting in more accurate and context-aware document analysis.

Traditional text clustering methods, such as model based text clustering [26] and feature based text clustering [27], which struggle to accurately categorize and group similar feedback due to their limitations in extracting the context and meaning of words and phrases. To address the limitations of conventional text clustering methods in the realm of deep text clustering, pre-trained deep contextual models, such as BERT [6], GPT-2 [10], and RoBERTa [8] were introduced and they have shown remarkable success in deep text clustering. These models learn contextualized representations of words, which capture the meaning and context of words in a sentence or document. However, the standalone contextual models (i.e. BERT, GPT-2 and RoBERTa etc.) are having certain limitations when it comes to deep text clustering. Some of these limitations include: Lack of clustering and interpretability abilities, Ambiguity in capturing document similarities, Limited domain knowledge and ability to handle noise and outliers.

In general, the standalone contextual models are designed for NLP tasks like text generation, classification, and sentiment analysis but not for deep text clustering on their own. Moreover the contextual model may produce latent representations in clustering, which are difficult to interpret and understand [28, 29]. As contextual models are trained on large amounts of generic data and they may not have specific domain knowledge relevant to the text being clustered, which leads to the suboptimal clustering results. The contextual models are optimized for generating coherent and fluent sentences and paragraphs in language modeling [30]. Hence they are unable to effectively handle the noise and outliers (irrelevant data points) in text data clustering. Due to the lack of a direct mechanism for modeling the document-level similarity [31], the contextual models may not be able to effectively capture the similarities between documents.

To overcome the limitations of standalone contextual models, this paper proposes an integrated deep contextual embedding model, in which the pre-trained contextual models are integrated with clustering algorithms, to increase the cluster quality and document classification accuracy. In this method, pre-trained models [32] are employed to extract high-level semantic features from text documentsand these extracted features are then used as input to the clustering algorithms [33] to group similar documents. Integration of clustering algorithms with pre-training contextual models will enhance the ability to capture the context and meaning of words and phrases along with the underlying structure and relationships in the text data. This integration will also help to interpret and understand the latent representations in clustering along with identifying and removing noise and outliers, leading to better clustering performance. The proposed model provides a more robust and generalizable approach to text clustering, reduces the need for manual feature engineering, and offers superior performance compared to traditional and standalone contextual text clustering models.

3.1 Deep contextual text clustering model

In the context of deep text clustering and document classification, the proposed model (Figure 1) for integrating pre-trained contextual models [6-8] with clustering algorithms is designed in several steps are: Dataset selection, Data pre-processing, word embedding’s, Positional encoding, Semantic Features Extraction, Feature vector dimensionality reduction, Text clustering and Document classification.

3.1.1 Data selection

As part of the “Integration of pre-trained contextual models with clustering algorithms” project for deep text clustering and document classification, we selected the popular text datasets for experimentation and evaluation.

Each dataset ‘D’ comprises multiple documents $\left\{d_1, d_2, d_3 \ldots d_n\right\}$ spanning different categories $\left\{c_1, c_2, c_3 \ldots c_n\right\}$, making them widely used benchmarks in NLP research for text classification and clustering tasks. These datasets are particularly well-suited for evaluating the performance of text clustering and classification algorithms due to their extensive size and diverse range of topics.

3.1.2 Pre-processing

After dataset selection process, the next step involves in preprocessing the text data with respective techniques [34], which is crucial to increase the cluster quality and document classification accuracy in further steps. As part of this, the unwanted characters or symbols from the text data (i.e. punctuation marks, emojis, special characters, and HTML tags etc.) are removed using the regular expressions. To make simplify the clustering process, the input text data d_i is converted into the input token (i.e., individual words or phrases) sequences. Frequently occurring stop words (i.e. “the”, “and”, “is”, etc.) are eliminated and the unique words are stemmed from the input token sequences to reduce the processing complexity.

23-1.png

Figure 1. Block diagram of the Deep Contextual Embedding Models based text clustering

 3.1.3 Word embedding

After pre-processing, different techniques are used to convert the text (tokens) into numerical representations for clustering. In case of our pre-trained contextual models, the input token sequences are converted into the “word embedding vectors” using a transformer-based neural network [35]. word embedding vectors is to represent words and phrases in a continuous, semantically meaningful space. These word embeddings are essential for capturing contextual information, understanding the relationships between words, and enhancing the accuracy of text clustering and document classification tasks. They enable the integration of pre-trained deep contextual models with clustering algorithms, providing a foundation for improved document analysis.

The word embedding’s [36] generated by the contextual models is used to extract high-level semantic features for text clustering and document classification.

Let's take a sequence of words, denoted as $W=\left\{w_1, w_2, w_3, \ldots w_n\right\}$, where ‘n’ represents the length of the sequence. Each word in the sequence is represented by a d-dimensional vector, denoted as, where d represents the dimension of the embedding space. The embedding for the $i^{t h}$ word in the sequence is denoted as $e_i=\left\{e_{i 1}, e_{i 2}, e_{i 3}, \ldots e_{i d}\right\}$.

3.1.4 Positional encoding

In our approach, we incorporate the order of words in the input sequence by implementing positional encoding [37]. By employing this technique, we are able to encode details regarding the position of each token in the input text, which allows our model to capture the sequential information and contextual meaning of the words. The positional encoding is a vector that is added to the input embedding’s before they are fed into the multi-head attention mechanism. The formula for the positional encoding is:

$P E_{i, 2 j}=\sin \left(\frac{i}{10000^{\left(\frac{2 j}{d_{\text {model }}}\right)}}\right)$       (1)

 $P E_{i, 2 j+1}=\cos \left(\frac{i}{10000^{\left(\frac{2 j}{d_{\text {model }}}\right)}}\right)$     (2)

where, 'i' is the position of the word in the input sequence, 'j' is the index of the dimension in the embedding vector, and $d_{\text {model }}$ is the dimension of the embedding vector. This equation is to incorporate the positional encoding $P E_{(i, j)}$ to the generated embedding’s for $i^{t h}$ word and $j^{t h}$ dimension is defined as:

 $P E_{(i, j)}= \begin{cases}\sin \left(\frac{i}{10000^{2 j / d}}\right), & \text { if } 'j' \text { is even } \\ \cos \left(\frac{i}{10000^{2 j / d}}\right), & \text { if } 'j' \text { is odd }\end{cases}$           (3)

The positional encoding values are now added to the original embedding vector for each word to create the final embedding vectors, which has to be processed by the subsequent layers of the proposed model.

At this moment, the contextual model has to generate a hidden state for each token in the input text, which captures the contextualized meaning [35] of the token based on its context within the sentence. For each token of the position encoded final embedded vector, our pre-trained contextual model generates a sequence of hidden states, $H=\left\{h_1, h_2, h_3 \ldots h_n\right\}$, where each $h_i$ is a d -dimensional vector representing the hidden state at position‘i’. The hidden states are generated by iteratively applying multi-head self-attention and feed-forward layers, with each layer utilizing the output of the previous layer to produce a more refined representation of the input text.

3.1.5 Semantic features extraction

When applying pre-trained contextual models [6, 9] in the context of deep text clustering with clustering algorithms [26], multi-head self-attention is utilized to capture the high-level semantic features of each word in the input text. The multi-head attention mechanism [38] is considered as a type of the self-attention mechanism where instead of using a single set of learned matrices to generate queries, keys, and values, multiple sets of these matrices are used. The attention computation is performed multiple times in parallel, each time using a different set of matrices. The ultimate output is achieved by concatenating the results of each attention head and then passing them through a linear transformation.

The pre-trained contextual model generates word embedding’s along with positional encodings, which are then fed as input to the multi-head self-attention layer.The word embedding’s (E) in self-attention are fed through learnable weight matrices W_Q, W_k  and W_v of the linear transformation to create the queries (Q), keys (K), and values (V) vectors as:

 $Q=W_Q * E, \quad K=W_K * E$ and $V=W_V * E$          (4)

These vectors are then split into multiple heads (head₁,head₂,…head_n), and each head computes the attention score between the queries and keys to obtain the attention weights. The output of each attention head is obtained by computing a weighted sum of the values using the attention weights. The hidden states produced by the multi-head self-attention mechanism in our model can be represented as:

 $M_{-}Head (Q, K, V)= Concat \left(\right.head _1, . ., head\left._n\right) W^O$      (5)

where, head $_i={Attention}\left(Q W_i^Q, K W_i^K, V W_i^V\right)$, and $W_i^Q \in \mathfrak{R}^{d_{\text {model }}{ }^* d_k}$, $W_i^K \in \mathfrak{R}^{d_{\text {model }} * d_k}$, $W_i^V \in \mathfrak{R}^{d_{\text {model }}{ }^{* d_v}}$, $W^O \in \mathfrak{R}^{h d^* d_{\text {model }}}$.

In this equation, $Q, K, V \in \mathfrak{R}^{n_v^* d_{\text {model }}}$ represents the input query, key, and value matrices, where $d_k$ denotes the dimensions of the key and value vectors for each head, ‘n’ represents the count of attention heads, and $d_{\text {model }}$  represents the dimensionality of the input embedding’s. Multi-head attention model is to capture complex semantic features and relationships within the text data. Multi-head attention allows the model to focus on different parts of the input text simultaneously, enabling it to extract diverse and informative contextual information. This is crucial for improving the performance of text clustering and document classification, as it helps in understanding the nuances, nuances, and subtle patterns in textual data, ultimately leading to more accurate results. In Multi-Head attention model the attention for each head is evaluated as:

 $\operatorname{Attention}(Q, K, V)=soft$_$max\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$         (6)

In this context, the dimensionality of the key vector is denoted as $d_k$, and a scaling factor $\sqrt{d_k}$ is applied to prevent the dot product from becoming too large, which can impact the soft-max function. The soft-max function is then applied to the dot product of the query Q and key vectors $K^T$ along the second axis to obtain the attention weights. The resulting attention weights are then multiplied by the value matrix V to obtain the final output.

3.1.6 Feature vector dimensionality reduction

The output of the contextual model may result in high-dimensional feature vectors. To enhance computational efficiency and reduce the dimensionality of these vectors, dimensionality reduction techniques like Principal Component Analysis (PCA) [39] or t-Distributed Stochastic Neighbor Embedding (t-SNE) [40] can be employed. The choice of dimensionality reduction technique should depend on the data characteristics and analysis objectives. In this study, we have opted for the linear technique, and PCA is used for implementing the dimensionality reduction process.

PCA is a widely used linear method for reducing dimensionality. It converts high-dimensional data into a lower-dimensional space while preserving the maximum possible variance in the data. This dimensionality reduction technique ensures that the clustering algorithms can operate more efficiently and effectively. The reduced feature vectors obtained through PCA can be used for further analysis such as clustering or classification. Let ‘X’ be the original high-dimensional feature vectors of shape (n_samples, n_features). The initial process in PCA is to center the data ‘$X_C$’ by subtracting the mean from each feature:

 $X_C=(X-\bar{X})$         (7)

After centering the data, the covariance matrix S can be calculated with number of samples n and mean of the centereddata $X_C$ as:

 $S=\frac{1}{n-1}\left(X_C\right)^T *\left(X_C\right)$        (8)

The eigenvectors (v) and eigenvalues (λ) of the covariance matrix can be computed as:

 $S v=\lambda v$            (9)

The top k eigenvectors can be selected based on the highest k eigenvalues. The data can then be projected onto the new basis formed by these top k eigenvectors as shown in below:

 $X_R=X W$          (10)

where, X is the original data, W is the matrix of top k eigenvectors, and $X_R$ is the transformed data with reduced dimensionality.

3.1.7 Text clustering

Following dimensionality reduction, clustering algorithms such as K-means, Hierarchical Clustering, or DBSCAN can be applied to group similar documents based on the extracted high-level features [33]. K-means clustering [41] is a widely used unsupervised clustering algorithm that partitions data points into K clusters based on their similarity. In the context of our contextual-based text clustering, the feature vectors obtained from the pre-trained contextual models after PCA reduction are utilized as input data points for clustering. The k-means algorithm can be defined as follows:

(1) Initialization: Randomly select K cluster centroids from the data points.

(2) Assignment: Allocate the data points with the closest centroid.

(3) Update: Update all centroids by computing the data points mean of them.

(4) Repeat: Continue with steps 2-3 until convergence is achieved.

The Elbow Method [41] involves running K-Means for a range of K values and calculating the sum of squared distances (inertia) of data points to their assigned centroids for each K. A plot of K against inertia is created, and the "elbow point" where the inertia starts to level off is considered as the optimal K. Silhouette analysis measures how similar each data point in one cluster is to the data points in the same cluster compared to the nearest neighboring cluster. For each K value, a silhouette score is calculated, and the K that results in the highest silhouette score is chosen as the optimal number of clusters. In the case of k-means clustering, the distance between two data points is frequently defined using the Euclidean distance metric [41], which can be expressed as:

 $d\left(x_i, x_j\right)=\sqrt{\sum_{k=1}^n\left(x_{i, k}-x_{j, k}\right)^2}$         (11)

In above context the $x_i$ and $x_j$ are two data points, n is the dimensionality of the feature vectors ($X_R$), $x_{i, k}$ and $x_{j, k}$ are the corresponding ‘k’ components of the obtained feature vectors for $x_i$ and $x_j$, respectively. The objective of k-means clustering is to minimize the total sum of squared distances between data points and the centroids to which they are assigned, as expressed by the following equation:

 $\sum_{i=1}^N \sum_{j=1}^K w_{i j} d\left(x_i, c_j\right)^2$          (12)

where,‘N’ is considered asthe count of data points, K is clusters count value, $c_j$ is the centroid of the $j^{t h}$ cluster, and $w_{i j}=1$, if data point $x_i$ is assigned to cluster j, or 0 otherwise.The k-means algorithm converges when the cluster assignments no longer change, i.e., the centroids no longer move. The ideal number of clusters, denoted as K, can be determined through techniques such as the elbow method or silhouette analysis.

3.1.8 Document classification

After the k-means clustering [41] is performed on the lower-dimensional feature vectors, the resulting clusters can be treated as different classes or categories for document classification. At this moment, we perform the document classification process on each cluster by assigning a label to the entire cluster. The best way to do this is by computing the centroid [42] of each cluster and using it as a representative vector for that cluster. Then, for each document in the cluster, we can compute its similarity to the centroid vector using a similarity metric such as cosine similarity. The similarity score can be used to assign the document to the appropriate class. The equation for computing the centroid vector for a cluster C is:

 centroid $(C)=\frac{1}{|c|} \sum_{i \in C} X_i$          (13)

where, C is considered as the documents count in the cluster, and $X_i$ represents the feature vector of document i. Once the centroid vector is computed, the comparison score amongst the document $X_i$ and its centroid vector $c_C$ can be calculated using cosine similarity:

 $\operatorname{similarity}\left(x_i, \mathbf{c}_{\mathbf{C}}\right)=\frac{x_i \cdot \mathbf{c}_{\mathbf{C}}}{\left|x_i\right|\left|\mathbf{c}_{\mathbf{C}}\right|}$         (14)

where, ‘.’ represents the dot product amongst the two vectors, and ‘|.|’ represents the Euclidean model of a vector. Based on the obtained similarity scores, we can assign each document in the cluster to a class with the highest score. In this way the total documents are classified based on the contained cluster similarity scores.

The key objectives of this study were to improve text clustering and document classification accuracy by integrating pre-trained Deep Contextual Embedding Models (DCEMs) like GPT-2 with clustering algorithms. The methods involved generating contextual embeddings, applying multi-head self-attention, and dimensionality reduction. The overall findings demonstrated that the integrated approach outperformed traditional methods, achieving higher accuracy in text clustering and document classification on datasets like AG News, Reuters-21578, and IMDB reviews.

These findings from the experimental evaluation offer valuable insights into the effectiveness of various combinations of contextual embedding models for text clustering and document classification, which can be used to enhance the performance of text clustering algorithms. Experiments proven that, the ability of DCEM based clustering algorithms are effectively captured the semantic information in text data, coupled with the power of deep learning techniques, make them a potent tool for clustering and classification tasks in various domains.

A potential future research direction is to explore semi-supervised or unsupervised methods that reduce the reliance on large labeled datasets for deep contextual model training. Another direction is to investigate the potential benefits of incorporating domain-specific knowledge into Deep Contextual Embedding Models to improve their performance on domain-specific text data.