An Interpretable Topic-Aware Deep Learning Framework for User Experience Analysis in Electronic Wallet Reviews

TM, or topic detection, is an automated tool for discovering topics from textual data. This process is one of the important steps in understanding and organizing unstructured textual data. These topics are implicitly represented by a set of words. Given a set of textual data, the topic detection problem is how to find these sets of words [17]. Currently, there are several methods that can be used to extract topics in a set of textual data, for example, NMF [18] and latent Dirichlet allocation (LDA) [19]. NMF is a matrix factorization method by factoring matrix A into two matrices W and H, all of whose elements are non-negative. If A is a word-per-data matrix, then W is interpreted as a word-per-topic matrix. In other words, the columns of W are sets of words interpreted as topics. Each topic is usually represented only by the most frequently occurring words. Several simulations have shown that NMF provides better accuracy than other methods [20]. TM (LDA and NMF) is very effective for topic exploration, but is descriptive and not designed for continuous automatic classification of new data.

Most research in topic detection focuses on its application in real-life applications, for example, detecting topics in Urdu on Twitter [21] and monitoring trending topics on Instagram [22]. Most studies stop at topic interpretation, without bridging the results to an operational predictive system.

Another major application is using topics for news article classification and clustering [23]. Other papers place topics as features. Analysis of fintech features has been carried out using topic LDA models [24, 25]. In the study [26], the focus is on sentiment classification and analyzing accuracy using two-class and three-class sentiment analysis datasets. TM approaches offer interpretability, while DL models offer accuracy, but the literature tends to treat the two separately. This results in fintech analytics systems often facing a dilemma between explainability and performance, without a unified framework that balances the two.

NMF aims to approximate a matrix V (data matrix) by the multiplication of two non-negative matrices, W and H. In the context of topic processing, V is a document-word matrix formed from a set of text documents, where rows are documents and columns are words or features [27]. NMF is structurally robust, but is not designed for end-to-end classification. The basic formula of NMF is as follows:

$V \approx W \times H$            (1)

where,

$V \in R^{m \times n}$ is the document-word matrix with dimension $m$ $\times n$ (number of documents m and number of words $n$).

$W \in R^{m \times k}$ is the basis matrix, where each column represents the distribution of words from one topic. Dimension $m \times k$, with $k$ as the number of topics.

$H \in R^{k \times n}$ is the topic-document representation matrix, where each row shows the weight or importance of a topic in a document. Dimension $k \times n$, with $k$ as the number of topics.

2.1 Non-negative matrix factorization optimization

The NMF optimization process is to find the values of W and H that minimize the error function (cost function), which is formulated as the Frobenius norm of the difference between V and WH (Eq. (2)):

$\min _{W, H \geq 0}\|V-W M\|_F^2=\sum_{i=1}^m \sum_{j=1}^n\left(V_{i j}-(W H)_{i j}\right)^2$            (2)

where,

$\|\cdot\|_F^2$ is the Frobenius norm, which computes the square root of the sum of all elements in the matrix.

The non-negativity condition $W, H \geq 0$ ensures that all elements in matrices W and H remain non-negative.

2.2 Update rules for non-negative matrix factorization

NMF is solved using iterative multiplicative update rules. The update rules for W and H are as follows:

Updates for the matrix H:

$H=H \odot \frac{W^T V}{W^T W H}$             (3)

Updates for the matrix W:

$W=W \odot \frac{V H^T}{W H H^T}$             (4)

where,

$\frac{A}{B}$ is the element-by-element division between two matrices A and B.

2.3 Topic processing with non-negative matrix factorization

The V matrix is generated from the document representation (using Term Frequency–Inverse Document Frequency (TF-IDF)). The V matrix is decomposed into W and H matrices. The W matrix gives the distribution of words in topics, where each column represents one topic, and the H matrix shows the importance of topics in the document. Once the W and H matrices are obtained, we can extract topics by identifying the words with the largest weights in the W columns.

This CNN–LSTM model combines the ability of CNN to capture local features of text and LSTM to capture temporal or sequence dependencies in text, which are then classified using a Dense layer with softmax. Here is the mathematical representation of CNN–LSTM in the given code:

2.3.1 Embedding layer

The embedding layer maps tokens or words into fixed-dimensional vectors.

X is an input token with sequence length T. $W_e \in R^{v \times d}$ is the embedding matrix, where V is the vocabulary size and d = 256 is the embedding dimension. Then the embedding of each word can be written as:

$E(X)=W_e . X$             (5)

where, X is a sequence of words in one-hot vector form, and the result is a dense representation $E(X) \in R^{T × d}$.

2.3.2 Conv1D layer (CNN)

The convolutional layer captures local features from the input embedding. The convolution operation can be formulated as:

$F(t)=\operatorname{ReLU}\left(\sum_{i=1}^k W_{\text {conv }}(i) \cdot E(t+i-1)+b_{\text {conv }}\right)$             (6)

where,

$W_{\text {conv }} \in R^{k \times d \times f}$ is the convolution kernel with $k=5$ as the filter size and $f=128$ as the number of filters. $E(t)$ is the input embedding at timestep $t$.

$b_{\text {conv }}$ is the bias. The ReLU activation function is applied elementwise.

2.3.3 MaxPooling layer

MaxPooling layer performs pooling to reduce dimensionality and extract the most important features.

$P(t)=\max (F(t: t+p-1))$

where, p is the pooling window size, in this case 2.

2.3.4 Dropout

Dropout is a regularization technique used to prevent overfitting by ignoring some neurons during training with a certain probability.

$D(t)=P(t) \cdot m(t)$

where, m(t) is a binary mask that has a value of 1 with probability 0.3 (dropout rate), and 0 otherwise.

2.3.5 Spatial dropout

Spatial dropout is similar to regular dropout, but dropout is applied spatially to the entire feature vector in a timestep, rather than individually to neurons. Where $r_t$ is a binary mask applied over the entire timestep:

$S(t)=D(t) \cdot r_t$

by randomization with a dropout probability of 0.3.

2.3.6 Bidirectional LSTM

Bidirectional LSTM is used to capture temporal information from two directions: forward and backward. For LSTM, the basic operation at each timestep t can be formulated as:

Forget gate: $f_t=\sigma\left(W_f \cdot\left[h_{t-1}, x_t\right]+b_f\right)$,

Input gate: $i_t=\sigma\left(W_i \cdot\left[h_{i-1}, x_t\right]+b_i\right)$,

Candidate cell state: $\acute{C}_t=\tanh \left(W_c \cdot\left[h_{t-1}, x_t\right]+b_c\right)$,

Output gate: $o_t=\sigma\left(W_o .\left[h_{t-1}, x_t\right]+b_o\right)$,

Cell state update: $C_t=f_t \odot C_{t-1}+i_t \odot \acute{C}_t$,

Hidden state update: $h_t=o_t \odot \tanh \left(C_t\right)$.

For Bidirectional LSTM, there are two calculation sequences: one for the forward direction and one for the backward direction. The output is combined:

$h_t^{\text {bi }}=\left[h_t^{\text {forward }}, h_t^{\text {backward }}\right]$             (7)

There are two LSTM layers, where the first LSTM returns the complete sequence, and the second LSTM returns the final output of each sequence.

2.3.7 Fully connected layer (dense layer)

The Dense layer converts the LSTM output into class output by using the softmax function for classification:

$\begin{gathered}z=W_d \cdot h+b_d \\ y=\operatorname{softmax}(z)\end{gathered}$             (8)

where, $W_d$ is the weight, $b_d$ is the bias, and $z$ is the linear score for each class.

2.3.8 Loss function (categorical crossentropy)

The loss function is used to measure the error in predicting the probability of the actual class:

$\mathcal{L}=\sum_{c=1}^C y_c \log \left(\hat{\mathrm{y}}_c\right)$             (9)

where,

$\mathcal{L}$ is the loss function, in this case categorical cross-entropy.

$\sum_{c=1}^C y_c \log \left(\hat{\mathrm{y}}_c\right)$ is the sum for each class $c$ from 1 to $C$, where $C$ is the total number of classes.

$y_c$ is the actual label for class $c$. Its value is 1 if the true class is $c$, and 0 otherwise.

$\hat{y}_c$ is the predicted probability of the model for class $c$, which is the output of the softmax function.

This function calculates how much the predicted label $y_c$ differs from the true label $y_c$ for all classes, and then penalizes it based on the logarithm of the predicted probability. If the predicted $y_c$ is close to 1 for the true class, then the value of $\log \left(\hat{\mathrm{y}}_c\right)$ will be large and negative, smaller (i.e., a smaller loss penalty). Conversely, if the predicted $y_c$ is close to 0 for the true class, then $\log \left(\hat{\mathrm{y}}_c\right)$ will be large in negative, and a larger loss penalty.

Random Over-Sampling with Neighborhood Cleaning (ROS–NCL) is used to improve robustness and avoid bias towards important minority issues. ROS–NCL is responsible for balancing without creating artificial linguistic patterns and maintaining the integrity of text representation. Although various sampling techniques exist to address data imbalance, ROS is the preferred method for handling imbalanced data [28]. The majority class has been shown to be susceptible to misrepresentation, demonstrating that each dataset is uniquely sensitive to imbalance, a problem that can be solved using ROS [29]. A combination of oversampling and undersampling methods can be one solution in resolving imbalance cases [30]. The undersampling method used is Neighborhood Cleaning Rules. NCL can improve the identification of minority classes that are difficult for classifiers to recognize [31].

A unified framework that is both interpretable and robust does not yet exist. TM approaches provide an interpretable thematic structure but remain largely descriptive and inadequate for scalable predictive applications. Machine learning (ML) models, while highly accurate, often lack semantic transparency and offer limited insight into the specific dimensions of user experience that drive classification results. Furthermore, the inherent class imbalance in fintech reviews introduces bias toward majority topics, reducing robustness in detecting minority but high-risk issues. Therefore, a unified framework that integrates interpretability, predictive power, and imbalance-aware learning remains underexplored.

4.1 Dataset analysis

This study uses a dataset of comments from Fintech e-wallet users in Indonesia, collected in the period 2023–2024 from the Play Store using API technology. A total of 42,680 rows in the dataset have been cleaned, with the following breakdown: DANA: 8,834; DOKU: 2,159; GoPay: 6,829; I-Saku: 4,821; LinkAja: 2,951; OVO: 7,942; ShopeePay: 7,204; Tapcash: 1,940. The dataset has been cleaned using a process involving language filtering, duplicate removal, text normalization, and the removal of entries containing fewer than four words.

Table 1 shows the results of the NMF algorithm with 8 topics generated, with words that have weights. The weight of the word shows how important or dominant the word is in each topic. Words with higher weights have stronger relevance to the topic. This table is useful for understanding what users pay most attention to in each topic discussed in the review, helping in further understanding user preferences.

Table 1. Topics with resulting weight

Table 2. Keyword and user comment

Table 3. Expert validation results

Table 4. Meaning of the topics

Table 5. Number of topic clusters

Table 2 shows examples of keywords arranged from a collection of sentences contained in the content column. Table 2 presents the results of extracting eight topics using NMF from the e-wallet user comment dataset. The topic ID column displays the numerical identity of the NMF extraction results. The Keywords column represents the distribution of dominant words in user comments on each topic, obtained from the H matrix. The Text column represents an example of an e-wallet user comment document with the highest probability weight in the W matrix. This demonstrates a direct relationship between the distribution of words within a topic and the distribution of topics within documents, making the displayed text an empirical representation of the keyword pool within that topic.

Table 3 presents the results of the topic quality validation for the NMF TM output, based on assessments by three experts using three criteria: semantic relevance, keyword consistency, and domain suitability, on a scale of 1–5. The evaluation results indicate that the eight generated topics possess interpretability, keyword consistency, and suitability for the fintech context, making them appropriate for further analysis of e-wallet user reviews. Domain suitability received the highest score, demonstrating that the topics align well with the e-wallet analysis domain. Keyword consistency and semantic relevance also achieved high scores, indicating that the keywords constituting each topic are semantically related and effectively represent the topic's meaning.

Table 4 helps understand the primary focus of each user review based on the most relevant keywords and reflects important aspects such as ease of use, cost, performance, and security of fintech applications. This meaning-making process was carried out by developing prompts. The meaning-making results using OpenAI were validated by two fintech experts and one language expert.

Table 4 continues the explanation from Table 2, adding columns for topic name and meaning, showing the results of topic analysis conducted using OpenAI to identify the meaning of keyword combinations in user reviews of fintech applications. Each topic contains keywords that frequently appear together in reviews and is assigned meanings based on the context in which they are used.

Table 5 shows the amount of data in each topic cluster, with varying amounts per topic. The data shows that the amount of data in each topic varies significantly. This supports the use of an imbalanced data algorithm.

4.2 Analysis of convolutional neural network–long short-term memory

Table 6 shows that the accuracy rate is above 95%, indicating that fintech user comments on topics such as ease of use of the application, low-cost transfers, and application performance are relatively consistent and easily predicted by the model. Focused comments and frequent repetition of the same pattern make it easier for the model to identify keywords and contexts of each comment. The high error rate for Topic 6: this class consists of user complaints or more complex problems, such as refunds, failed transactions, or account limitations, which can be expressed in various ways by users. High variability in comment content can make it difficult for the model to classify comments correctly. Although the data imbalance handling technique has been carried out, the accuracy rate cannot be significantly improved. However, from several papers on measuring the level of multi-variable accuracy, an accuracy of 75% is acceptable for use.

Table 6. CNN–LSTM–ROS–NCL accuracy

Overall, the CNN–LSTM–ROS–NCL model performs very well in most classes except class 6, which has problems with variability in the data. Classes with simpler or more general content, such as those related to transactions and ease of use, are easier for the model to predict, while classes related to complaints and technical issues, such as class 6, still require more specific handling to improve their classification performance.

Table 7 shows that the total accuracy of the model is 0.96; this shows that overall, the model works very well in classifying the data. Macro average has precision 0.96, recall 0.94, F1-score 0.95: The macro average value calculates the average of each topic individually. Weighted average: precision 0.96, recall 0.96, F1-score 0.96: The weighted average calculates the average by considering the amount of support in each topic. This value is more representative of the overall performance because it takes into account the size of each topic. Since topics with greater support have high performance, the weighted average is very good. The macro average value is lower because Topic 6 affects the overall recall value, while the weighted average remains high because topics with large support have good performance. The overall model is very accurate, with precision, recall, and F1-score close to 1 in most topics. This shows that the model is very effective for classification in most topics, with some improvements likely needed to improve performance in topics with less data or that are more difficult to recognize.

Table 7. Accuracy CNN–LSTM–ROS–NCL

4.3 Comparative analysis

Table 8 shows the comparison of the accuracy levels of three different models, namely CNN–LSTM–ROS–NCL, LSTM–ROS–NCL, AND CNN–ROS–NCL, on eight different classes. Each model combines DL techniques (CNN–LSTM) with methods to handle data imbalance, namely ROS–NCL.

Table 8. Comparison of several algorithms

Table 9. CNN–LSTM without data balancing

Table 10. Classification results of the Random Forest algorithm

The CNN–LSTM–ROS–NCL model overall has the highest accuracy in most classes, indicating that the combination of CNN–LSTM enhanced with ROS–NCL provides the best performance on the data faced. CNN–ROS–NCL shows quite competitive performance, with little difference in accuracy compared to CNN–LSTM–ROS–NCL, but still superior to LSTM–ROS–NCL in some classes. LSTM–ROS–NCL generally has the lowest performance among the three models, especially in handling class 6, where this model only achieves an accuracy of 65.93%.

The CNN–LSTM algorithm without a balancing method achieved an accuracy of 77% with a macro F1-score of 0.75, indicating that the model was affected by class distribution imbalance. After applying ROS–NCL, accuracy rose to 96%, while the macro F1-score improved to 0.95. This improvement demonstrates that ROS–NCL effectively reduces classification bias and enhances recognition capabilities across various topic categories, as shown in Tables 7 and 9. A comparison of Table 9 and Table 7 reveals that the F1-score increased with the use of the balancing algorithm. The most significant improvement was observed in Topic 7, where the F1-score rose from 0.61 to 0.96. This indicates that ROS–NCL successfully improved representation learning for previously underrepresented topics. However, Topic 6 remained a challenge, with the recall value dropping from 0.88 to 0.74 following the balancing process.

Table 10 presents an analysis of the Random Forest algorithm, showing a macro-average precision of 0.599. This value indicates that the model is capable of generating topic predictions with a reasonable level of accuracy based on dominant word patterns within the documents; however, it does not outperform the CNN–LSTM and CNN–LSTM–ROS–NCL algorithms.

Table 11. Example of classification error analysis

Table 12. Accuracy test results of several algorithms

Class 6 in Table 8 is the most difficult to predict by all models, as seen from the significant drop in accuracy compared to other classes. The CNN–LSTM–ROS–NCL model performs best in this class with an accuracy of 77.19%, far above both LSTM–ROS–NCL (65.93%) AND CNN–ROS–NCL (60.06%). This may indicate that class 6 has a highly imbalanced distribution or has features that are difficult for the model to learn, so LSTM in combination with CNN helps capture important temporal patterns in the data. The CNN–LSTM–ROS–NCL combination is the most effective approach in dealing with data imbalance issues and in predicting difficult classes. Although CNN–ROS–NCL performs closely, the use of LSTM provides additional benefits in dealing with more difficult classes, such as class 6. The LSTM–ROS–NCL model performs the worst, especially on difficult classes, indicating that using only LSTM without CNN may not be powerful enough to capture the complexity of the data.

The analysis of misclassifications regarding Topic 6 in Table 11 reveals that the low recall is attributable not only to linguistic variation but also to semantic overlap with several adjacent topics. For instance, reviews concerning missing balances during transfers exhibit characteristics of both account/balance issues and transfer-related problems. Similarly, reviews regarding failed cash withdrawals involve a service-related context yet explicitly feature cash withdrawal activity that strongly points toward the digital payment topic. Reviews concerning login failures and the loss of app features may even encompass multiple service-related intents simultaneously. These findings indicate that a single user review can contain several interconnected intents, whereas the research framework assigns only one dominant topic. Consequently, Topic 6 possesses more complex semantic boundaries compared to some of the other categories.

CNN–LSTM–ROS–NCL gives the best result with 96.26% accuracy, but the difference with CNN–ROS–NCL is only 0.05 percentage points, which means that in this dataset, the temporal pattern captured by LSTM is not very significant in improving the performance, as seen in Table 12. CNN–ROS–NCL shows that CNN with proper data imbalance handling is very effective in predicting the results. LSTM–ROS–NCL shows lower performance, indicating that CNN may be more suitable for this dataset than using LSTM alone, especially when the data has a more dominant spatial pattern. Overall, the combination of CNN with imbalance handling methods (ROS–NCL) gives the best performance, while LSTM only provides a little additional benefit in some cases.

The integration of ROS–NCL proved effective in reducing bias toward the majority class while maintaining classification stability across topics. The combined CNN–LSTM architecture further improved the model's ability to capture local textual features and sequential dependencies, contributing to an overall accuracy of 96.26%. While the additional contribution of LSTM compared to CNN alone was small on this dataset, the hybrid architecture demonstrated stronger robustness on more complex classes.

The comparison of accuracy between CNN–LSTM–ROS–NCL and CNN–ROS–NCL shows an improvement of only about 0.05 percentage points, indicating a limited contribution; however, CNN–LSTM provides improvements for more difficult classes, particularly class 6, thereby demonstrating its benefits for more complex language patterns.

From a practical standpoint, the proposed topic-based classification framework utilizing NMF algorithms and a CNN–LSTM–ROS–NCL classification model provides structured information on user experience categories that can serve as a basis for analyzing fintech service improvements. By organizing user feedback into easily understandable semantic categories and linking them to predictive analytics models, this framework enables fintech organizations to more systematically understand the distribution patterns and characteristics of issues arising in user reviews.

The authors would like to express their gratitude to the Ministry of Higher Education, Science, and Technology for the financial support provided through the Regular Fundamental Grant Program for the 2024 Fiscal Year, under Decree Number 0459/E5/PG.02.00/2024 (dated May 30, 2024) and Contract Number 108/E5/PG.02.00.PL/2024-011/LL6/PB/AL.04/2024. Appreciation is also extended to Soegijapranata Catholic University for the facilities and support provided throughout the course of this research.