A Hybrid Text Summarization Approach Using Neural Networks and Metaheuristic Algorithms

Text summarization is the process of generating a concise summary from a lengthy text document. It can be classified into extractive and abstractive summarization. Text summarization techniques have many applications such as generating news headlines, search snippet generation, and simplifying access to information [1].

To solve the complex problems in text summarization, bio-inspired metaheuristic algorithms and neural networks can be applied [2]. We propose using a Multi-hidden Recurrent Neural Network (MRNN) to generate a feature vector for each sentence. A novel mayfly-harmony search (MHS) algorithm is then applied to optimize the feature weights for selecting the most relevant sentences [3].

To illustrate the advantages of the text-MRNN-MHS-text summarization technique, we conduct tests on three benchmarks Document Understanding Conference (DUC) datasets [4]. Studies reveal our method performs better than several extractive summarizations and cutting-edge abstractive baselines [5]. A good summary is based on the ability of demonstration and comprehension, whereby empirical evidence also supports the importance of MRNN-MHS of concepts and events in representing and understanding documents [6].

The objectives of this study are:

• To extract informative summaries from multiple documents using textual features and metaheuristic algorithms
• To propose a hybrid MRNN-MHS text summarization approach that combines neural networks and optimization algorithms [7].

The main contributions of this study are:

• A MRNN model to generate sentence feature vectors.
• A MHS algorithm for optimizing feature weights.
• Evaluation on benchmark DUC datasets showing competitive performance [8].

An automated multi-document summarising method creates a condensed and comprehensive document from numerous articles. The three steps of the multi-document summarising process include pre-processing, generating feature vectors, extracting features, and producing summaries. An overview of the summarising technique is shown in Figure 1. Many documents are submitted into the summary system, including Docu1, Docu2..., and DocuN. The initial pre-processing of the documents yields the output, which is then passed through feature extraction, feature vector generation, and summary representations to produce the final summary.

The types and lengths of the input documents are varied for the multi-document summarizing task, and they can be loosely categorized into three groups:

Stop word removal, stemming, and sentence splitting make up the pre-processing stage of our suggested solution. The stop-words removal removes the words that are most frequently used but don't have a clear meaning, including articles, prepositions, conjunctions, interrogatives, assisting verbs, etc. Inflected terms stem by going back to their original form. Each sentence in a document is distinguished by sentence breaks [13].

The pre-processed multi-documents are combined in this section by eliminating unnecessary sentences and ensuring that all relevant topics are covered. The sentence resemblance is first designed as shown below (1).

$S({{t}_{k}},{{t}_{k'}})=\alpha \times {{S}_{WMD}} \;({{t}_{k}},{{t}_{k'}})+(1-\alpha )\times {{S}_{NGD}} \;({{t}_{k}},{{t}_{k'}})$      (1)

1.png

Figure 1. Proposed method of MRNN-MHS

where,

${{S}_{WMD}}({{t}_{k}},{{t}_{k'}})=\frac{\sum\nolimits_{{{x}_{q}}\in {{t}_{k}}}{\sum\nolimits_{{{x}_{q'}}\in {{t}_{k}}}{,{{S}_{WMD}} \;\;\, \left( {{x}_{q}},{{x}_{p'}} \right)}}}{|{{t}_{k}}|.|{{t}_{k'}}|}$       (2)

And

${{S}_{NGD}}({{t}_{k}},{{t}_{k'}})=\frac{\sum\nolimits_{{{x}_{q}}\in {{t}_{k}}}{\sum\nolimits_{{{x}_{q'}}\in {{t}_{k}}}{,{{S}_{NGD}}\left( {{x}_{q}},{{x}_{p'}} \right)}}}{|{{t}_{k}}|.|{{t}_{k'}}|}$        (3)

$x_q$ and $x_{q^{\prime}}$ are terms in $t_k$ and $t_{k^{\prime}}$, which are two phrases, respectively.

We have employed a linear grouping of WMD and NGD-founded comparison functions to evaluate the similarities between two condemnations and examine how these functions affect the efficacy of the summarization strategy.

Reducing redundancy and increasing coverage are the goals of single document creation. To account for this, sentences more similar to the document than the average sentence similarity is regarded as having high coverage. We express it mathematically as follows:

$S({{t}_{k}})+S\left( {{t}_{k'}} \right)\ge \frac{2}{m'}\sum\limits_{k=1}^{m'}{S({{t}_{k}})}$      (4)

If there is less than a threshold value $P \in(0,1)$ of similarity between two sentences, they are measured to be non-redundant condemnations. The non-redundancy criterion for two curses is distinct below.

$S\left( {{t}_{k}},{{t}_{k'}} \right)\le P$       (5)

Divide Inequality (4) by Inequality (5) to get (6)

$\frac{S\left( {{t}_{k}} \right)+S\left( {{t}_{k'}} \right)}{S\left( {{t}_{k}},{{t}_{k'}} \right)}\ge \frac{\frac{2}{m}\sum\nolimits_{k=1}^{m'} \;{S({{t}_{k}})}}{P}$       (6)

The sentence pairs are chosen to be a part of the single document creation if they satisfy inequality (6).

3.1 Text features

The sentences in Document D are graded in this part based on various text properties. Give the text feature x the symbol $f_y$. The score of the manuscript feature for condemnation is then shown by $R f_x$. The following provides a succinct description of the text topographies employed in this work.

Position of Sentence: It is a crucial component of sentence extraction. Usually, a manuscript's opening and last condemnations contain more important info than the rest. We compute the score as shown in Eq. (7) to account for this factor.

$R{{f}_{1}}\left( {{t}_{k}} \right)=\frac{|N/2-k|}{N/2}$       (7)

Length of the Sentence: The concise and very long sentences are filtered out using this function. The following procedure can be used to get the average distance of a batch of condemnations.

$AV(R)=\frac{\min \left( |{{t}_{k}}| \right)+\max \left( |{{t}_{k}}| \right)}{2}$       (8)

where, AL(S) is the document's average sentence length, and S is the collection of sentences. The equation is used to determine the phrase length score using the average size (9)

$R{{f}_{2}}\left( {{t}_{k}} \right)=1-\frac{|AV(R)-|{{t}_{k}}||}{\max \left( |{{t}_{k}}| \right)}$        (9)

Sentence Similarity: Discovery of the typical info in the sentences will allow you to score them. It might be stated that it views sentences with more shared information as being more pertinent.

$R{{f}_{1}}\left( {{t}_{k}} \right)=\frac{\sum\nolimits_{k'=1,\,k\ne k'}^{|{{d}_{j}}|} \;\; {\left( S\left( {{t}_{k}},{{t}_{k'}} \right) \right)}}{|{{t}_{k}}|}$         (10)

Sentence with Title Words: Sentences with title words highlight the most pertinent sentences in a document. The formula for calculating this score is provided below.

$R{{f}_{3}}({{t}_{k}})=\frac{\sum\nolimits_{W\in {{t}_{k}}} \; {\left( Count\left( gra{{m}_{L}} \right) \right)}}{|W|.|{{t}_{k}}|}$      (11)

where, |W| represents the entire amount of title arguments, W represents the title term.

Sentence with Numerical Data: In general, sentences with numerical data point to vital information that will probably be included in the summary. The calculation looks like this.

$R{{f}_{4}}({{t}_{k}})=\frac{\sum\nolimits_{nd\in {{t}_{k}}} \;\; {\left( Count\left( gra{{m}_{L}} \right) \right)}}{|nd|.|{{t}_{k}}|}$       (12)

where, nd stands for the mathematical information and |nd| for all the numerical data combined.

The Number of Proper Names: An adequate noun designates a particular individual, location, or institution. It is the one trait that may express more information in a phrase than any other. The equation looks like this.

$R{{f}_{5}}({{t}_{k}})=\frac{\sum\nolimits_{PN\in {{t}_{k}}} \;\; {\left( Count\left( gra{{m}_{L}} \right) \right)}}{|PN|.|{{t}_{k}}|}$         (13)

where, "PN" stands for "proper noun."

Sentence with Frequently Occurring Words: Often occurring words can also describe the document's title, which contains essential information. The top 30% of this work's most often occurring terms are regarded as frequent words. We divided the entire amount of frequent arguments by amount as the whole of views to normalize them. The calculation looks like this.

$R{{f}_{6}}({{t}_{k}})=\frac{\sum\nolimits_{fw\in {{t}_{k}}} \;\; {\left( Count\left( gra{{m}_{L}} \right) \right)}}{|fw|.|{{t}_{k}}|}$       (14)

where, fw stands for the familiar words.

Sentence Significance: Sentence significance provides information on the contribution of the sentence. Briefing the importance of each word part of the judgment yields the phrase's significance score. Thus, the tf-idf approach is used to determine the extent of arguments. The following formula is used to determine a sentence's relevance score.

$R{{f}_{7}}\left( {{t}_{k}} \right)=\frac{\sum{Sig({{x}_{qk}})}}{|{{t}_{k}}|}$        (15)

where,

$Sig({{x}_{qk}})=f{{e}_{q}}\times \log \frac{m'}{{{m}_{q}}}$        (16)

Here, Sig(x_qk) denotes the importance of the word x_q in the condemnation t_k. The amount of sentences in np is x.

A MRNN is used to produce a feature vector. Several text qualities with their optimum weights were used to score the phrases in the text summary task to identify the pertinent ones. This ensures that the system-generated summaries are accurate. This technique may produce a text feature vector exceptionally effectively. By interpreting the relevant portion at each generational step, an attention-encoder will provide some conditioning to ensure that only the input words are focused. This method allows for creating a shorter version of a given statement while maintaining its meaning.

The Mayfly algorithm and Harmony search are combined to get superior results. This technique entailed extracting several solutions from diverse districts of the search space using the Mayfly Algorithm and processing them with Harmony Search. Cosine similarity and statistical traits are also used in the hybrid Mayfly Algorithm and Harmony Search (MHS) algorithm. In additional arguments, a mixture of four data analyses and two cosine similarity-based metrics was used as the impartial purpose to assess each sentence's quality, to be maximized during the evolution process. As a result, the recommended approach's summarising problem is a maximizing problem, and the MHS aims to select the original text phrases with the top scores.

The purpose of immediate illustration is to produce instantaneous document sets containing valuable info. The most effective sentence selection strategy chooses the key sentences that convey the summary by associating the condemnation revealing score produced by the optimization procedure with deference to a prearranged threshold assessment.

3.2 MRNN to generate a feature vector

Recurrent Neural Networks (RNNs) are viewed as neural networks with bidirectional data flow in contrast to Multi-Layer Perceptrons, which receive information from lower layers and feed it to developed ones. From early processing steps to earlier processing stages, the data flow propagates. We implement Jeff Elman's proposed small Recurrent Neural Network in this investigation. The model depicted in Figure 2 makes use of three network layers. Context neurons record the output of each hidden neuron at a time (s - 1) and then send it back to the hidden layer at a time (s). Context neurons continuously hold copies of the hidden neurons' initial values until a parameter-updating rule is functional at the period due to propagation across the repeat connections (s - 1). (s - 1). As a result, a set of the state summarising earlier inputs is retained and acquired by the network model.

2.png

Figure 2. A structure of RNN

The perceptron model used in the RNN-based neural network comprises two hidden layers and two setting layers (one framework for each hidden layer). What follows is a survey of the perfect: As shown in the study [14], the network consists of 18 input neurons, ten first-hidden neurons, ten first-context neurons, ten second-hidden neurons, and one output neuron. There are no differences between the neurons in the first and second context layers, which are concealed (see calculations lower).

$CT_{l}^{1}(t)=hd_{q}^{1}(s-1)$        (17)

$C T_l^1(t)$ equals $h d_q^1(s-1)$ if it is the lth neuron in the first situation layer at period s.

$CT_{m}^{2}(t)=hd_{g}^{2}(s-1)$       (18)

$C T_m^2(t)$ either equal or indicates the lth neuron in the second contextual layer at period $t$.

In the preceding example, $h d_g^2(s-1)$ acted as the $\mathrm{j}^{\text {th }}$ neuron in the second hidden layer. The feed-forward to the first hidden layer can be expressed in the following way:

$hd_{q}^{1}(s)=f\left( \sum\limits_{p}^{I}{v_{pq}^{1}{{x}_{p}}(s)} \right)+f\left( \sum\limits_{1}^{Co{{n}^{1}}}{u_{pq}^{1}CT_{1}^{1}(s)} \right)$       (19)

$f(net)=f\frac{1}{1+{{e}^{-net}}}$       (20)

where, $f$ (net) is an experimental stimulation purpose in which each hidden neuron at the hidden layers uses both Sigmoid and Softamax. $v_{p q}^1$ and $u_{p q}^1$ represent heaviness associates between the first hidden layer $h d_q^1(s)$ and the input layer $x_p(t)$, and between the first hidden layer $h d_q^1(s)$ and the first context layer $C T_1^1$, respectively.

The feed-forward to the second hidden layer can be stated as follows:

$hd_{g}^{2}(s)=f\left( \sum\limits_{q}^{HD1}{v_{qg}^{2}hd_{q}^{1}(s)} \right)+f\left( \sum\limits_{m}^{Co{{n}^{2}}}{u_{mg}^{2}CT_{m}^{2}(s)} \right)$         (21)

where, $v_{q g}^2$ and $u_{m g}^2$ denote weight associates among the second hidden layer $h d_g^2(s)$ and the first layer $h d_q^1$, respectively, and among the second hidden layer $h d_g^2$ and the second setting layer $C T_m^2$.

The following is one possible formulation for the feed-forward to the output layer:

${{O}_{k}}(s)=f\sum\limits_{g}^{HD2}{{{w}_{gk}}hd_{g}^{2}}(s)$          (22)

where, $w_{g k}$ stands for the weighted link among the second hidden layer $h d_g^2(s)$ and the output layer $O_k(s)$.

Also, this model's training aims to achieve the highest classification with the lowest Mean Square Error (MSE), which measures the difference between the upgraded RNN with GWO's projected output and the intended result. The MSE is determined as follows:

$MS{{E}_{pt}}=\frac{1}{n}{{\sum\limits_{k=1}^{n}{({{O}_{k}}(s)-{{d}_{k}}(s))}}^{2}}$      (23)

where, $\mathrm{n}$ denotes the amount of production neurons and $d_k(s)$ and $O_k(s)$ signify the $\mathrm{k}^{\text {th }}$ neuron's intended and actual outputs. The overall MSE through all examples is given as surveys:

$Tota{{l}_{MSE}}=\frac{1}{n}\sum\limits_{pt=1}^{T}{MS{{E}_{pt}}}$         (24)

where, pt  is an example design, and T is the number of training configurations.

By processing numerous Mayfly Algorithm results from various search space areas through Harmony Search, improved solutions were discovered by combining the Harmony Search and Mayfly Algorithm.

The class of insects known as Palaeoptera includes the order Ephemeroptera, which provides for mayflies. These insects are known as "mayflies" since they mostly appear in May in the UK. Before they are prepared to rise to the superficial as mature mayflies, immature mayflies spend numerous centuries developing as aquatic nymphs. A few meters above the water, most adult males congregate in swarms to appeal to the females. They customarily dance for their wedding, going up and down while establishing a beat. For mating, female mayflies travel to these swarms. After a short period is spent mating, they dump their eggs into the water to resume the cycle [15].

The MA enhancements improve the method's performance for both small and large-scale feature sets. The following list includes the MA components:

Male Mayfly Movement: Eq. (25) changes a male mayfly's location as follows:

$y_{m}^{t+1}=y_{m}^{t}+u_{m}^{t+1}$       (25)

where, $y_m^t$ represents the male mayfly's current position, and $y_m^{t+1}$ represents the position after adding $u_m^{t+1}$ respectively. The male mayflies have remarkable velocity and always have insufficient patterns overhead the water's surface. Eq. (26) calculates the speed of a male mayfly as follows:

$\begin{align}  & u_{kn}^{t+1}=g*u_{kn}^{t}+{{b}_{1}}*{{e}^{-\beta {{r}^{2}}_{p}}}\left( pbes{{t}_{kn}}-y_{kn}^{t} \right) \\ & +{{b}_{2}}*{{e}^{-\beta {{r}^{2}}_{g}}}*\left( gbes{{t}_{n}}-y_{kn}^{t} \right) \\\end{align}$       (26)

where, $u_{k n}^t$ is a mayfly's velocity in measurement $\mathrm{j}$ at period $\mathrm{t}$, $y_{k n}^t$ is its position at period t, $b_1$ and $b_2$ are optimistic attraction coefficients used to measure the contributions of the cognitive and social apparatuses. Correspondingly, $g$ is a gravitational constant and secure discernibility constant that maximizes a mayfly's discernibility to other mayflies. The finest position that a specific mayfly, $k$, has ever visited is represented by pbest $_k$, while the $j^{\text {th }}$ element of the location of the greatest male mayfly is represented by gbest $_n$. Since this is a minimization problem, the following changes are made to pbest $_k$ :

$pbes{{t}_{k}}=\left\{ \begin{align}  & y_{k}^{t+1} \\ & if\,\,\,fitness\,(y_{k}^{t+1})<fitness(pbes{{t}_{k}}) \\\end{align} \right.$         (27)

where, fitness $\left(y_k^{t+1}\right)$ offers the solution's quality, or the position's fitness value. Last but not least, $r_p$ is the Cartesian expanse among $r_g$ and pbest $_k$ while $r_g$ is the expanse among $r_g$ and gbest. According to the calculations shown in Eq. (28).

$|{{y}_{k}}-{{Y}_{k}}|=\sqrt{\sum\limits_{j=1}^{n}{{{({{y}_{kn}}-{{Y}_{kn}})}^{2}}}}$      (28)

where, $Y_k$ either represents pbest $_k$ or gbest and $y_{k n}$ indicates the location of the $\mathrm{j}^{\text {th }}$ component of the $\mathrm{k}^{\text {th }}$ mayfly. The algorithm's stochastic component depends on the best mayflies at a specific period continuing to complete the nuptial dance. This dance is mathematically represented in Eq. (29).

$u_{kn}^{t+1}=g*u_{kn}^{t}+d*r$       (29)

where, r is an arbitrary amount among € [1, 1] and d is the nuptial dance constant. As $d_{i t r}=d_0 \times \delta^{i t r}$, the nuptial dance constant progressively reductions. It is the current iteration count with a random value between € [0, 1], where d₀ is the starting assessment of the nuptial dance constant.

Female Mayfly Movement: Female mayflies seek out males and fly near them during mating. The Mayfly is currently located where:

$x_{m}^{t+1}=x_{m}^{t}+u_{m}^{t+1}$     (30)

where, $x_m^t$ represents the female mayfly's present location at period $t$, which is updated by its velocity, $u_m^{t+1}$. The superiority of the present explanation influences men's and women's attraction to one another; as a result, the best-performing woman is drawn to the best-performing guy, and so on. The equation is used to update a female's velocity (31).

$u_{kn}^{t+1}=\left\{ \begin{align}  & if\,\,fitness({{x}_{k}})>fitness({{y}_{k}}) \\ & g*u_{kn}^{t}+{{b}_{2}}*{{e}^{-\beta {{r}^{2}}_{mf}}}*(y_{kn}^{t}-x_{kn}^{t}) \\ & else\,\,if\,\,fitness({{x}_{k}})\le fitness({{y}_{k}}) \\ & g*u_{kn}^{t}+fl*r \\\end{align} \right.$        (31)

where, $u_{k n}^{t+1}$ is the location of the female mayfly $k$ in measurement $\mathrm{j}$ at period $\mathrm{t}, y_{k n}^t$ is the $\mathrm{j}^{\text {th }}$ section of the location of the male mayfly $k$ at period $t$, and $x_{k n}^t$ is the $\mathrm{j}^{\text {th }}$ constituent of the velocity of the $\mathrm{k}^{\text {th }}$ female mayfly's velocity at period $t . r$ is a arbitrary number between $[1,1]$ and $r_{i f}$ is the Cartesian expanse among male and female mayflies, which is assumed in Calculation 4. $\mathrm{a}_2$ and are the before distinct attraction continuous and visibility measurement, correspondingly. $g$ is the gravity coefficient previously distinct in Calculation 2. In the situation where a female is not involved to a man, $f l$ is an arbitrary walk coefficient, and $f l_{i t r}=f l_0 \times \delta^{i t r}$. Here, itr are two variables from Calculation that have already been defined.

Mayfly Crossover: The crossover operation requires the selection of a male and a female mayfly. The assortment of breeding partners is founded on suitability assessment, with the best male mating with the best female. Equation illustrates that a crossover results in the birth of two offspring (32 and 33).

$offspring1={{r}_{of}}*male+(1-{{r}_{of}})*female$          (32)

$offspring2={{r}_{of}}*female+(1-{{r}_{of}})*male$          (33)

As a result, the male replaces the male mayfly parent, the female returns the female mayfly parent, and $r_{o f}$ is a constant number between 0 and 1. The offspring's initial velocities are established to 0.

Mayfly Effect: Newly-born offspring are altered to improve the algorithm's exploratory capabilities. The offspring's variable is supplemented by a regularly distributed random number, as explained in

$offspring_{n}^{'}=oggsprin{{g}_{n}}+k$          (34)

where, k, the randomly chosen value, has a normal distribution.

Harmony Search

In the natural world, a specific connection exists between several sound waves with various frequencies. Humans frequently find comfort in musical performances. Its aesthetic evaluation determines the extent of the calming impact.

A musician aspires to the highest aesthetic state comparable to the world's best. As a result, choosing the optimal aesthetic condition is an optimization challenge with a predetermined objective function. The usual resonances produced by tools connected simultaneously estimate aesthetic value. Practice can raise the artistic value of the sound produced. The HS pseudo code is shown in Algorithm 1.

MHS Algorithm

The MA discussed in paragraphs III-A was initially established for ongoing optimization issues. In Algorithm 2, the redesigned FS algorithm known as MA-HS is described. Each linear combination in MA is first reduced to its binary form or to 0s and 1s before being assessed. This conversion is accomplished by using the S-shaped transmission purpose. This purpose provides the likelihood of selecting a specific feature in an explanation vector. It is a dependable purpose utilized by numerous investigators in the past. Figure 3 demonstrates the S-shaped transmission meaning employed in this approach.

3a.png

3b.png

Figure 3. S-shaped transfer function for binary representation of the continuous Mayfly search space

$S(y)=\frac{1}{1+{{e}^{-y}}}$      (35)

Eq. (35) updates the agent's feature during the conversion process.

$P_{d}^{t+1}=\left\{ \begin{align}  & 1\,\,\,\,if\,\,S({{P}^{t+1}})>rand \\ & 0\,\,\,if\,\,S(P_{d}^{t+1})\le rand \\\end{align} \right.$       (36)

where, $P_d^{t+1} \mathrm{~d}$ is the agent's updated feature subset, and is an arbitrary amount among 0 and 1, and $S\left(P^{t+1}\right)$ is the S-shaped transmission purpose as earlier distinct in Eq. (36). Figure 4 depicts the proposed framework's general layout.

4.png

Figure 4. A description of the MHS algorithm, which is used to solve FS problems

Fitness Function

In this section, the algorithm assesses the effectiveness of an explanation. Since it is a wrapper-based solution, a learning procedure has been employed. The classification accuracy is determined using the Multi-hidden Recurrent Neural Network (MRNN) classifier. The number of features and the classification error make up the fitness function. The authors of FS want to decrease the number of characteristics while improving accuracy simultaneously. Accuracy isn't employed in this case; categorization error is. This is so that the inaccuracy and the number of topographies can be concentrated. Thus, merging these two will lower the fitness purpose to a single impartial purpose. Figure 4 shows the flowchart of MHS. The formula for calculating the value of a feature subset is found in Eq. (37).

$\downarrow Fitness=\gamma \times \lambda +(1-\gamma )\times \frac{|f|}{|F|}$       (37)

where, $|f|$ is the amount of topographies in the feature subsection, $|F|$ is the amount of topographies in the supplied dataset, $\gamma \in[0,1]$ is a limitation that indicates the comparative influence of the organization error on the number of topographies, and $\lambda$ is the classification error.