Using Deep Learning to Generate and Classify Thyroid Cytopathology Reports According to The Bethesda System

ABSTRACT


INTRODUCTION
Attempts to integrate natural language processing (NLP) methods in areas where only classical approaches have existed for a long time become increasingly evident.The use of new approaches is justified by the fact that classical methods often do not give the desired results and require constant improvements.Therefore, modern natural language text processing technologies are being introduced increasingly often into areas not previously covered by them.
One of these areas is medicine, where natural language is used to record individual results.Having the fixed signs of the disease described by the doctor in natural language, one needs to solve the issues of classifying the disease according to the accepted scale and automatically generating the text of the medical report indicating all parameters used for subsequent treatment.
Let us consider the formulation and solution of the tasks of working with medical texts written in natural language in the case of classification and generation of a medical opinion on the description in the field of cytological studies of the thyroid gland.
The initial data are: -The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC) [1,2] according to which a thyroid nodule fine needle aspiration (FNA) material is classified into one of six diagnostic categories with different corresponding malignancy risk (ranging from 4 to 97%), that helps to determine patient further management tactics.
-A corpus of pairs of real texts (1.description and 2. Conclusion-a TBSRTC category) made by cytopathologists of the National Medical Research Center for Endocrinology.However, in this case, in many pairs, the first element (a description) is missing due to its placement in another field along with the diagnostic category.
It is necessary to build a sequence of mathematical models (pipeline) for the formation of a medical opinion according to the description indicating the Bethesda category.The results of the work may be of interest as a part of a thyroid cytopathology automated analysis system and can be used to generate a description and the corresponding category for a FNA sample.
The task in its general formulation in different fields of activity was solved by several research groups.Therefore, it is reasonable to consider the closest approaches to the classification and generation of text tokens and problems of time complexity, spatial storage, and processing of raw text arrays.
The related works are discussed further in section 2. Section 3 outlines the proposed approach to the automatic generation of Bethesda categories.In section 4, the results of the conducted quality studies of the proposed approach are discussed, and in section 5, similar works are described.Finally, section 6 summarizes the results of the study with conclusions and possibilities for its development.
This study primarily aims to innovate in the field of medical text processing by developing deep learning models tailored for Russian-language thyroid cytopathology reports.We specifically focus on utilizing Sequential and Transformer neural networks to classify diseases and generate comprehensive medical reports.The scope of this research is carefully defined to include an in-depth analysis of these neural network models, particularly assessing their performance in handling datasets of varying sizes.

LITERATURE REVIEW
Currently, several review papers offer text classification algorithms or put forward interesting ideas for evaluating medical text corpora.Thus, in the research [3], the use of complex approaches based on autoencoders and the use of convolutional networks for the classification of medical texts is discussed.The idea of mixing the received and generated data during learning as a method of improving the quality of the final model is interesting.This is important for us since to increase the body of raw data of missing medical descriptions, we have to solve the task of augmenting them based on available medical reports.
In the research [4], which focuses on learning text classification algorithms from scratch and obtaining the best representation only on similar data, approaches to obtaining training results on raw data are described, as opposed to widespread approaches to retraining architectures like BERT (Bidirectional Encoder Representations from Transformers).
The authors of the papers [5][6][7] involve architectures based on sequential and transformer networks to solve analytical medical problems, which can be applied to our task for the transfer of knowledge in the cross-domain field.
One of the tasks of this work is the classification according to a cytopathology description text, i.e., the formation of the Bethesda category [8] using deep learning.Using new approaches for text preprocessing and model training, both canonical transformers and their modifications, we will try to increase the accuracy of predictions relative to a given target threshold.
The solution to this problem lies in the way of creating a pipeline for preprocessing data and extracting key information from raw medical text corpora.The pipeline model is both a classifier of the Bethesda target label and a generator of a medical opinion on the medical description of the patient's problems.
The structure of this study is presented in the form of a sequence of sections.The first section describes the approaches in the preprocessing of the source data.In the second, a basic statistical model is created, according to which more advanced architectures will be evaluated.The following sections describe sequential approaches aimed at producing a short conclusions (diagnostic categories) based on a cytopathological description.Next, the transformer technologies and their training options are discussed.In conclusion, the results are summed up, and the validation of the resulting models is performed.
In the realm of medical text processing, various techniques have been explored in the literature.Sequential neural networks, known for their simplicity and effectiveness in small data scenarios, have been widely used for structured data analysis.They excel in tasks where data points are interdependent, such as time-series analysis.On the other hand, Transformer neural networks, a more recent development, are noted for their ability to handle larger datasets efficiently.Their architecture, based on self-attention mechanisms, allows them to process data points in parallel, making them particularly suitable for complex tasks like language translation and large-scale text generation.The literature also highlights hybrid approaches, combining the strengths of different models to optimize performance.These comparisons underscore the importance of choosing the right technique based on the specific requirements of the dataset and the task at hand.

Data preparation
Working with medical reports on the classification of thyroid nodules raises the task of marking and clearing data.The corpus of real data used in the study contains more than 27 thousand thyroid cytopathology reports gathered in the Laboratory of Cytology and Cytogenetics of the Department of Pathomorphology of the Center for Endocrinology over ten years of work (2013)(2014)(2015)(2016)(2017)(2018)(2019)(2020)(2021)(2022)(2023).The data was received in a blinded form (without personal information).
Medical texts have a weak internal structure, which leads to problems of clearing data from unnecessary tokens and highlighting the basic information necessary for further generation of signs.
There may be several Bethesda labels in one medical description.For example, "p11-1: uninformative material: peripheral blood cells, thyrocytes were not detected in the smear against the background of a thick colloid (according to Bethesda Thyroid Classification, category I)"; further in the same description: "l11-2: a node of colloidal to varying degrees proliferating goiter with cystic changes has been punctured (according to Bethesda Thyroid Classification, category II)".
Solving problems in the order of allocation of the target Bethesda label and key description tokens on an untagged set of data, we place the main emphasis on the primary use of regular expressions and probabilistic search for the most frequently encountered sentence tokens.To search for this pattern, a regular expression of the type '[IV]+' is used, which helps to find all possible combinations of the Bethesda label in the considered text.Sentences from the original data corpus, after clearing the data with the generated labels, can be grouped by class, getting the percentage of occurrence (Table 1).In this table, the general target field is presented after splitting groups of complex text queries into basic ones (with one class label), which expands the original groups with a class zero, without affecting the distribution of Bethesda labels previously affixed by doctors.
Part of the process of preprocessing raw data to obtain a more voluminous object space is the search for the number of sentences in the text and the probability of words appearing in each sentence, which is a hyperparameter for various types of data augmentation and validation of the obtained medical assessment reports.At the preprocessing stage, long text conclusions of patients, who had gone through multiple FNA due to having a multinodular goitre and therefore had several conclusions (Bethesda diagnostic categories), were divided.When dividing the data into logical elements, where each element was a sentence with a Bethesda label, the feature field expanded by 7.5%.
Further, after separating the classifier labels by cytopathological descriptions and diagnostic categories, the necessary tokens were allocated.
After cleaning and creating the resulting dataset with separated Bethesda labels in the range from 1 to 6, we obtained completely separated data to solve the problems of classifying the Bethesda label and generating a diagnostic category based on the medical description of the thyroid problem.Examples of cytopathological descriptions are presented in Table 2.

Cytopathological Descriptions
Label The material is not informative enough: Single dystrophic thyrocytes were found in thick smears with an abundant admixture of erythrocytes 1

In the smears, the cytogram is characteristic of a colloidal, to varying degrees proliferating B-cell goiter with areas of adenomatosis and cystic degeneration 2
In the smear with a heterogeneous cellular composition, among the abundance of erythrocytes and colloid, there are both groups of polymorphic thyrocytes with signs of goiter transformation and clusters of enlarged epithelial cells of irregular shape, with, sparse chromatin, densely located in shapeless and papillary-like structures 3

Numerous isolated groups of thyroid papillary cancer cells have been found in the smear, forming trabecular and papillary structures with single intra-row pseudoinclusions 6
The preprocessed source data (the main keyword tokens and the cleaned raw data) were saved in .csvformat.
Ethical expertise.The protocol of the study was reviewed and approved by the local Ethical Committee of the National Medical Research Center for Endocrinology (Protocol No. 14 dated 25.07.2023).

Creation of a statistical model
When building a complex system for classification according to Bethesda and generating a diagnostic category based on a cytopathological description, it is necessary to be sure that the model can be effective in terms of temporal and spatial complexity.
In the previously reviewed papers, CNN (convolutional neural networks) and LSTM (long short-term memory) networks with modifications of the original algorithm were used to solve these problems.Such approaches work well for the task but are difficult to interpret and also slow, which leads to time costs.
Basic models are suitable for estimating temporal and spatial capacities, which will be emphasized in the validation of the final architecture in the future.
To solve problems step by step, one wants to combine the embedding approach (a numerical vector derived from words) and clustering methods to determine the class label.This choice helps to optimize the query execution time [9].In this study, we use FastText-based approaches [5] and methods for normalizing the final vector representations.
The first method we will consider uses document token norms and clustering using the K-means algorithm.This method allows us to quickly determine the Bethesda label.After normalization and training of the algorithm, we make predictions and compare them with the target labels of the test dataset.
The algorithm divides the set of vector space elements into a pre-known number of clusters k.The main idea is that at each iteration the center of mass is recalculated for each cluster obtained at the previous step, then the vectors are divided into clusters again in accordance with which of the new centers turned out to be closer according to the selected metric.The algorithm ends when at some iteration there is no change in the intra-cluster distance.This happens in a finite number of iterations, since the number of possible partitions of a finite set is finite, and at each step the total square deviation decreases, so looping is impossible.
In the proposed method, at each step, we select the largest cluster to maximize the f-beta-score metric as a harmonic mean between completeness and accuracy, which will allow us to obtain an average value between two significant metrics in the problem.Since there is no bijective mapping of clusters to previously obtained data in the K-means algorithm, the best set of labels of the target class is first assigned a label and then removed from the label pool.Not to lose information, we first consider the largest clusters, and then gradually move on to smaller clusters, since when a cluster is deleted, its label is lost.In our experiment we use a square partition matrix into seven clusters where we try to find the best non-parametric partition for the data.
This approach is characterized by a strong overlap of classes.Therefore, when assigning the same target label to several classes at the same time, we use a greedy algorithm for selecting features, and we get a distribution of labels by classes, as in Table 3.
After calculating FB (f-beta-score) using a formula of the harmonic mean: where, PR is the accuracy, Recall is the completeness, and B is a configurable parameter of the offset between accuracy and completeness.We see that after a cycle of training and prediction, the f-beta-score exceeds the threshold of 0.52 only in classes I and IV.This approach does not work well on the entire set of data, and changing the priorities of classes does not change anything since a significant number of points overlap each other.
When considering the clustering problem, we immediately note the impossibility of using the basic implementations of the density-based spatial clustering of applications with noise (DBSCAN) and ordering points to identify the clustering structure (OPTICS) algorithms [9].We are interested in the full field of predictions, and these algorithms do not have a parameter for configuring the number of clusters because they are based on the ideas of availability points.
At this stage, we wanted to obtain FB>= 0.91.As we did not obtain it using embedding texts, we moved on to more advanced methods from the NLP field.

Recurrent approaches
Recurrent neural networks (RNNs) are very popular today in processing and analyzing data sequences, such as texts, time series, and audio or video streams.These architectures contain a cyclic layer that allows one to transfer information from the previous step in time to the next and to remember and use the previous contextual information for making decisions in the present.Formally, RNN is defined by the following equations: is the input data at the current step, ℎ  is the hidden state at the current step,   is the output data at the current step, W and b are the matrices of weights and offsets, respectively, and h and y are activation functions for the hidden and output layers, respectively.
One of the NLP tasks, where RNNs have proven to be very effective, is to predict the diagnosis according to the description of thyroid disease.Using RNN, it is possible to analyze the results of the patient's studies and predict the diagnosis with high accuracy.A research team from the University of Leiden in the Netherlands conducted a study where they used an RNN to determine the diagnosis of thyroid diseases based on patient blood test data [9].The study showed that RNN demonstrated higher prediction accuracy compared to traditional methods.
Last year, another group of researchers from Stanford University and Google Brain developed a new architecture for solving NLP problems, including predicting a diagnosis based on thyroid analysis [10].In this study, RNNs were used in combination with an attention mechanism that allowed the network to focus on the most significant elements of the sequence.
The results of the study also showed the high efficiency of the proposed method.The use of RNNs in the task of predicting the diagnosis based on thyroid analysis data allows us to obtain more accurate results and significantly reduce the diagnostic time, which is also confirmed by the author of the article on the use of neural networks in healthcare [10].
One of the most common approaches is the use of RNNs with LSTM cells.LSTM cells allow one to save information about previous network states and store it for a long time.An LSTM cell has the following components.
The use of LSTM cells allows an RNN to store and use information about previous states for a long time, which makes it especially effective for processing sequential data with longterm dependencies [11].This is especially important when thyroid tests are collected several times over a certain period.
Another approach is the use of CNNs with 1D convolutions.This approach is especially useful when processing data that can be represented as time series, such as multiple thyroid assays over a certain period, which can be written as functions of time:

filters) -> Activation -> Pooling -> Straightening -> Fully connected layers(output size)
In this case, 1D convolutions allow us to identify important temporal signs, such as peaks and trends, and use them to predict the diagnosis [12].
Finally, there are hybrid models combining several different neural network architectures that can be more efficient than each of them individually [13].To illustrate this statement, a comparative analysis of the results of RNNs with LSTM cells, CNNs with 1D convolutions, and hybrid models combining both types of models was carried out.The results of the experiment, the purpose of which was to identify the most suitable solution for class label prediction are shown in Table 4.The table shows that the hybrid model, which combined RNN and CNN, showed the best results for all metrics.However, the CNN model with 1D convolutions came pretty close to the results and also showed great learning speed and ease of use.

Applying pre-trained models
Today, there are a large number of deep-learning models used for processing medical texts.The largest number of such models work in the English-speaking field of text analysis [14,15].However, some models have been pre-trained in Russian.To build such models, the well-known transformers BERT and RoBERTa (Robustly Optimized BERT Approach) were used, which had already been pre-trained on common-language corpora [16].
In this paper, two other models are used, pre-trained on the Russian-language Wikipedia, and additionally on a special Taiga data corpus created by the SberDevice team [17,18].Both of these models were further transferably trained on publicly available medical and biomedical texts.Due to such training, RuBioBERT and RuBioRoBERTa models were obtained [19][20][21].
For a better understanding of the work of RuBioBERT and RuBioRoBERTa, we recall the basic principle of BERT.It consists of the fact that when training a neural network, a word is masked not only at the end of a sentence but also inside it.Such a task is called a masked language modeling task [14].
This approach allows the neural network to simultaneously learn (learn token embeddings) in both directions, thereby achieving deep bidirectionality, which means that the model considers the context on both sides of the word [22].This architecture uses the attention mechanism, which helps to highlight the context and the relationship of words with each other.
The RoBERTa model is the same BERT model but with optimized hyperparameters dynamic word masking.During the pre-training, RoBERTa is trained only to predict the masked word, whereas, in the BERT architecture, the model was also pre-trained on the task of next sentence prediction, i.e., it predicted whether the second sentence in a pair of sentences was a continuation of the first one [16,23,24].
For the task of determining the relationship between a cytopathological description and assigned Bethesda category, both of these models were used to understand which of them worked better when solving the classification problem.To compare RuBioBERT and RuBioRoBERTa, the following model parameters were used (Table 5).The learning process in each of the models was recorded after every 1,000 steps and tested.As a result, the results of model training were obtained (Tables 6 and 7).
It can be seen from the tables that RuBioRoBERTa showed higher accuracy than RuBioBERT.
A significant difference between the RuBioRoBERTa model and RuBioBERT is a significant training time (3 times higher than the time of the other model) and a significantly greater weight of the model itself.Depending on the frequency of the necessary retraining, both the first and second models can be selected for classification according to Bethesda.

Text data augmentation
In the real corpus of 27,000 pairs of description and diagnostic category data used in this study, real descriptions are present only in 6,300 pairs.The description text can also be moved to the diagnostic category field.In the remaining data pairs, only a diagnostic category is present.
Considering that the quality of the model increases with an increase in the training sample, the task of augmenting the missing descriptions was additionally set.To do this, we trained a Russian-language T5-base model made by Sberbank to restore the text of a cytophatological description based on text of conclusion.
Augmentation (which took much more time than finetuning the T5-base model) increased the sample of pairs by almost 4 times to 24,600 examples.The analysis of the augmented data showed that: -the average number of words in the original (36.2) and augmented (51.9) texts of descriptions did not differ much; -the maximum number of words in the original texts was 177 and in the augmented texts 125.

Figure 2. Distribution of descriptions within topics in augmented descriptions
To analyze the properties of the source and synthetic data, thematic modeling of LDA (Latent Dirichlet Placement) was carried out using the genism library, where the initial class (the first category according to Bethesda) is marked with zero and further increases.The number of topics was taken equal to 6, according to the number of Bethesda categories.
The distribution was sorted by increasing probabilities to be able to see the general pattern.The distributions of the original and synthetic data are very similar (Figures 1 and 2).
In addition to the distribution of the descriptions by topic, the distribution of topics by words was constructed to highlight the top words describing a specific topic (Tables 8 and 9).The vocabulary in the topics almost does not match, but structurally the descriptions are similar.

RESULTS
Fine-tuning of the Russian-language T5-base model to solve the problem of generating a diagnostic category based on a description was carried out on augmented (24,600 examples) and initial (6,200 examples) samples.
Then, with the help of a trained model, predicted diagnostic categories were generated based on the available source descriptions, using regular expressions.According to the initial and predicted cytopathological descriptions, the Bethesda classes were obtained.
The accuracy of training and validation of the neural network model used for automatic classification of the disease description by Bethesda is shown in Figure 3.
To assess the quality of the created models of automatic generation of diagnostic categories on the text of the FNA description, a set of metrics for summarizing the text of ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (ROUGE-1, ROUGE-2, ROUGE-L) was used.
It is considered that the indicators ROUGE-2 and ROUGE-L are well suited for the tasks of abstracting individual documents, and ROUGE-1 and ROUGE-L show good results in evaluating short abstracts.Considering that the medical report can be both short and medium in length, we used all three indicators.ROUGE automatically generated descriptions with references or a set of references (human created descriptions).The key idea of the ROUGE family metrics is to estimate the intersection by n-grams between the source text and the generated one (in our case, between the text written by the doctor and the text generated by the model).The ROUGE-1 metric considers only individual words (unigrams), adhering to the bag of words philosophy, which does not allow evaluating the quality of word combinations.The texts of most of the generated medical assessment reports coincide with the texts of the medical assessment reports written by doctors.Examples of differing medical assessment reports are given in Table 10.Now we can conduct statistical tests for the two subproblems being solved, classification and generation.In the first case, the value will be f-score, in the second the resulting Rouge-2 values, the values are presented in Table 11.For classification values, we took the basic p-value threshold for text generation tasks, we set more stringent measures and take the threshold for accepting the null hypothesis equal to 0.08.

DISCUSSION
Methods of converting natural language texts for medical purposes are currently being actively developed.
The tasks being solved are difficult in terms of data processing and further training of the model since errors in the domains of medical tasks are critical.This forces us to collect data more carefully with an understanding of what results we want to achieve with augmentation using ready-made or trained models.Approaches to generating and evaluating data arrays are discussed in papers [9,10].
However, in our task, based on a corpus of ready-made texts, the main focus was on converting raw data and not on attempts to anonymize the personal data of patients, which had been done by other people in advance.The ideas of preprocessing formed the basis for obtaining data clusters but this did not show the expected tangible increase in target metrics in the future.
In the broader context of text classification, similar studies have encountered challenges related to preprocessing efficacy.For instance, the studies [25,26] faced a comparable situation where traditional preprocessing methods alone did not yield significant advancements in target metrics.This parallel underscores the inherent intricacies of text data, particularly in medical narratives, and the need for nuanced preprocessing strategies.
The main approaches that could help solve the problem were based on the sequential generation of matrix kernels in recurrent networks and architectures based on convolutions.These approaches are not new but necessary for generating the required labels of the class under study, and the ideas of the applied methods are partially covered in study [13,14].
The approaches proposed in the paper to solving the problems of data preprocessing, classification, and augmentation proved to be good at generating Bethesda labels but were not effective enough for complex token sequences.
This leads to an understanding of the issue of insufficient data in the training sample.To solve it, attempts were made to use ready-made pre-trained architectures with additional training on the target data set.Such techniques are based on the algorithms proposed in study [16].
Comparing our approach with similar studies in the field, we observed that [27] encountered a similar data insufficiency issue in their research.They addressed this challenge by employing a transfer learning strategy, achieving significant improvements in their model's performance.In our case, leveraging pre-trained architectures allowed us to navigate the data limitations, resulting in enhanced accuracy and robustness in handling complex token sequences.
To obtain the text of a complete description, as a result of generating a sequence of tokens of a medical opinion, previously pre-trained BERT models can be used.However, to expand the feature space, data augmentation and additional training of transformer models on the existing corpus of texts are required.This is what will allow for obtaining more accurate and complete medical reports at the output of the model.
In our study, we employed two primary models for the classification of thyroid cytopathology reports in the Russian language: Sequential neural networks and Transformer neural networks.Both models demonstrated commendable performance, but a closer examination reveals nuanced differences in their effectiveness.
The choice of model depended on the scale of the dataset.Sequential neural networks shone in resource-constrained scenarios, offering robust performance, while Transformer neural networks demonstrated their prowess when dealing with larger datasets, where their ability to handle complex token sequences yielded even higher accuracy.

CONCLUSIONS
This study addressed the challenge of processing Russianlanguage thyroid cytopathology reports using deep learning models.We utilized Sequential and Transformer neural networks, comparing their effectiveness in disease classification and report generation across different dataset sizes.
Our findings revealed that Sequential neural networks are more suited for smaller datasets, showing high accuracy and efficiency.In contrast, Transformer models demonstrated superior performance with larger datasets, excelling in complex text processing tasks.These insights are crucial for advancing the application of deep learning in medical text analysis, particularly in non-English contexts.
The result of this work was the creation of a pipeline.At its first stage, statistical methods were used to clean and preprocess raw data, which worked faster than neural networks with almost the same qualitative result.
Next, models of a transformer operating in two modes were trained as a Bethesda label generator and as a medical assessment report token generator based on the input description.
The proposed pipeline has an interface for preprocessing and retraining the model and can be converted into a serializable .pklfile.
Since the work was done in a high-level Python language, the final developments can be scaled.With the help of sklearn libraries to scale the interface and with the help of transformers to replace individual steps of the converter of the tasks being solved, the methods of preprocessing data from other medical fields can be left unchanged.
In terms of research metrics, an improvement of 2% was achieved, which is due to the proposed approaches to processing Russian-language data and combining methods of retraining ready-made architectures.
The practical significance of our results lies in their potential to make the classification of thyroid cytopathology reports easier, reduce manual labor, optimize resources, and pave the way for improved healthcare efficiency.
This work can be implemented in systems that rely on the automation of medical predictions and can help doctors make predictions much faster than using manual methods.
Considering the ethical component of this work, we note that the resulting pipeline is not yet a replacement for a doctor but is positioned as an assistant and interlocutor when forming a medical assessment report on the patient's problems.

Figure 3 .
Figure 3. Graphs of the accuracy of training and validation of the neural network model in classification

Figure 4 .
Figure 4. Graph of the loss function

Figure 5 .
Figure 5. Graphs of ROUGE summarization metrics This problem is partially eliminated by ROUGE-2 and ROUGE-L, as they evaluate matches by bigrams and the longest sequence of words, respectively.The last two metrics are more demanding to text generation than ROUGE-1.The change in the logistic loss function (loss) and the metrics of the ROUGE family depending on the step of the butch .
Suggested by the Doctor Generated L p/p 21.A benign nodule of the thyroid gland, diagnostic category II according to the criteria of the Bethesda classification system.L n/3 21.An atypical structure of unclear significance, diagnostic category III according to the criteria of the Bethesda classification system L n/3 21.An atypical structure of unclear significance, diagnostic category III according to the criteria of the Bethesda classification system.L n/3 21.Benign nodules of both lobes of the thyroid gland, diagnostic category II according to the criteria of the Bethesda classification system Cells of a neoplasm suspected of malignancy (papillary thyroid carcinoma?)Suspicion of a malignant neoplasm, diagnostic category VI according to the criteria of the Bethesda classification system A neoplasm suspected of malignancy, diagnostic category V according to the criteria of the Bethesda classification system A thyroid neoplasm of unclear malignant potential, diagnostic category IV according to the criteria of the Bethesda classification system

Table 1 .
Percentage occurrence of labels in the case

Table 3 .
Label clustering matrix

Table 4 .
Model comparison

Table 6 .
The RuBioBERT learning process

Table 7 .
The RuBioRoBERTa learning process

Table 8 .
Top tokens of the original descriptions