A Deep Reinforcement Learning Framework with Maximal Marginal Relevance-Based Reward Optimization for Query-Focused Extractive Multi-Document Summarization

Studies on automatic text summarization have developed significantly over the past decades. Current research can generally be classified into several directions, such as redundancy reduction techniques (e.g., MMR), QFS methods, and RL-based summarization.

2.1 Redundancy reduction technique

The paper [14] natural language processing as a knowledge-intensive (KI) model that eliminates the basis on pre-existing document groups. This system comprises two components: a summarization controller and a retrieval module. The retrieval module is Dense Passage Retrieval (DPR), which automatically searches using methods like BM25 for potentially relevant documents from a large-scale knowledge corpus. Then the summarization controller combines a multi-step process approach with a large language model (GPT-3.5). A few-shot prompting strategy to first extract query-relevant information and then generate a concise, non-redundant summary. The limitation of this study is that the model's performance decreases slightly when operating in a large open-domain corpus.

This research produces an interactive query-based summarizing system to enhance the relevance and quality of summaries for scientific documents [15]. The suggested system incorporates user interaction at several stages to enhance the summary, which integrates user participation by introducing a chance for the user to use the RAKE algorithm. Using the GA algorithm, the system can expand the chosen sentences; the final summary is created by aggregating the set of user-preferred and expanded sentences and ranking them using the MMR technique. It evaluated the effect of interaction with the user through various methods. 45 AI graduate students tested the proposed model by applying it to 72,000 scientific documents from the Web of Science (AI category).

2.2 Query-Focused Summarization

Yao et al. [16] introduced an RL system for QFS, which replaces conventional supervised learning with a policy-gradient training model, allowing the paradigm to enhance sequence-level rewards rather than token-level cross-entropy objectives. This method addresses the well-known incompatibility between RL and the teacher force, enabling stable gradient propagation across token sampling. The suggested system combines multiple reward signals—including a novel semantic similarity reward derived from custom passage embeddings trained via the Cluster Hypothesis and BLEU and ROUGE—which significantly optimizes semantic query relevance. This study is limited by several factors, including the need for foundational computational resources, such as multi-GPU hardware and long training times, which prevented comprehensive hyperparameter tuning, according to action-space exploration and trust assignment difficulties.

The RL training through long text sequences is still challenging. Zhang et al. [17] introduced the Qontsum, a contrastive learning model to optimize the relevance between the user’s query and the created summary. Different from conventional approaches, which are based on attention mechanisms only, Qontsum explicitly distinguishes between relevant and irrelevant document chunks. This framework faces several limitations; it cannot explicitly balance between relevance, redundancy, and difficulty in handling general or very specific queries. Also, this proposed model is based on contrastive fine-tuning but does not handle computational efficiency or adaptability in multi-document or large-scale QFS scenarios.

Lamsiyah et al. [18] proposed to leverage transfer learning from pre-trained sentence embedding models (uSIF, USE-DAN, and USE-Transformer) to represent documents’ sentences and users’ queries. Furthermore, BM25 and the semantic similarity function are linearly combined to retrieve a subset of sentences based on their relevance to the query. Finally, the MMR criterion is applied to re-rank the selected sentences by maintaining query relevance and minimizing redundancy. Experiments are conducted using three standard datasets from the DUC evaluation campaign (DUC 2005–2007). This proposed approach is limited by its reliance on fixed-length summaries (e.g., 250 words). The evaluation is limited to standard benchmark datasets (DUC 2005–2007), lacking sentence-level coherence modelling and the absence of fine-tuning the pretrained models. The linear combination of BM25 and semantic similarity scores, although effective, is manually weighted and lacks a learned optimization process that could better capture complex interactions between lexical and semantic relevance. In future work, I plan to investigate the potential of the newest models, T5 and GPT-3-3-3, for the text summarization task. And explore transfer learning abilities from pre-trained models for summary generation.

2.png

Note: DRL = Deep Reinforcement Learning, MMR = Maximal Marginal Relevance.

3.png

Figure 3. Reward components

The combination process between the selected sentence vector and query vector, resulting from the embedding stage, contributes to constructing an environment that simulates how an agent interacts to generate a summary. The constructing agent observes the current state represented by the integration of the current sentence (selected sentences or remaining candidates) embedding and the query embedding, allowing an agent to make context-aware selection decisions from the action space. The environment defines two potential actions: select or skip a sentence and transition through the document sequentially, updating the agent's state after every decision taken. During this process, the agent accumulates rewards until all sentences are processed or a maximum summary length is reached. This proposed framework enables the DQN agent to learn an optimal extraction policy that maximizes semantic quality and cohesion while minimizing redundancy, thereby distilling high-quality summaries based on queries. Reward function design is one of the most important contributions of the proposed system. Because it helps an agent learn how to select sentences that are coherent, most relevant, and non-redundant, corresponding to the query. The reward function is carefully constructed as a multi-component that balances three basic measures of summary quality that are relevance, coverage, and coherence, while penalizing redundancy, as shown in Figure 3. This model implements the MMR technique with a BERT-based coherence model and a coverage measure in building the reward function, as in Eq. (1), that will be given to an agent during training (at every time the DQN agent chooses a sentence, the reward will be calculated). Each part is weighted based on its contribution to the total summary quality.

$r_t=\underbrace{\left(\lambda_1 \times \text { Relevance }+\lambda_2 \times \text { Novelty }\right)}_{M M R \text { technique }}+\underbrace{\left(\lambda_3 \times \text { Coherence }\right)}_{\text {BERT-based coherence }}+\lambda_4 \times \text { Coverage }$                    (1)

where, $r_t$ denotes the reward value at time step $t . \, \lambda_1, \lambda_2, \lambda_3, \lambda_4$ are weighting parameters that control the contribution of each component in the reward function.

In this proposed system, the priority given to MMR (0.60) is to maintain relevance and diversity, while (0.25) is provided for BERT-based coherence to preserve the semantic fluency. Finally (0.15) assigned for coverage scores, ensuring the information integrity. The reward weights were assigned according to the proper significance for each component in QFS, prioritizing relevance and redundancy reduction, followed by coherence and coverage.

The $\lambda_1 \times \text { Relevance }+\lambda_2 \times \text { Novelty }$ represents the implementation of MMR, which aims to handle some challenges faced by generative extractive summarization tasks, such as relevance to the query (measured using cosine similarity between the query embedding and sentence embedding), as described in Eq. (2).

$\operatorname{Relevance}(S, Q)=\cos (\operatorname{embedding}(S), \operatorname{embedding}(Q))$                  (2)

where, S denotes the candidate sentence. Q represents the query.

Diversity (novelty) is achieved by reducing duplicates and ensuring that the selected sentences are distinct from those previously selected for the generated summary as illustrated in Eq. (3).

$ \operatorname{Novelty}\left(s, S_{\text {selected }}\right)=-\max _{s^{\prime} \in \text { Sselected }} \cos \left(\operatorname{embedding}(s), \text { embedding }\left(s^{\prime}\right)\right)$                 (3)

where, s represents the candidate sentence. $ S_{\text {selected}}$ denotes the set of sentences that have already been selected in the summary. $ s^{\prime}$ represents a sentence belonging to the selected summary set $ S_{\text {selected}}$. $\text {embedding}(\cdot)$ denotes the vector representation generated using the BERT model. $ \cos (\cdot)$ represents the cosine similarity function.

The MMR guarantees that the final summary is comprehensive and varied. It captures various aspects of the query without repeating information, as described in Eq. (4). This process ends when the required length of the generated summary is reached.

$ M M R(s)=\underbrace{\lambda \times \operatorname{Similarity}(s, Q u e r y)}_{\operatorname{Relevance}(S, Q)}-\underbrace{(1-\lambda) \times \max _{s^{\prime} \in S} \operatorname{Similarity}\left(s, s^{\prime}\right)}_{N o v e l t y(\text {news}, \text { Sselected})}$                   (4)

where, s represents the candidate sentence. Q denotes the input query. $ S_{\text {selected}}$ shows the set of sentences already selected in the summary. $ s^{\prime}$ is a sentence in the selected set $ S_{\text {selected}}$. $ Similarity (\cdot)$ denotes the cosine similarity between sentence embeddings generated using the BERT model. λ represents the balancing parameter controlling the trade-off between query relevance and redundancy reduction.

In QFS, guaranteeing that chosen sentences are both non-redundant and relevant is an essential objective; for this reason, it has the highest weight. During design, the reward function also takes into consideration the coherence component (0.25), introduced to encourage semantic consistency between consecutively chosen sentences, enhancing the logical flow and readability of the created summaries. The BERT-based coherence model in Eq. (5).

$ \operatorname{coherance}\left(s_{t-1}, s_t\right)=\frac{1}{n-1} \sum_{i=1}^{n-1} \cos \left(s_{t-1}, s_t\right)$                (5)

where, $s_t$ represents the current candidate sentence selected at time step t. $ s_{t-1}$ denotes the previously selected sentence in the summary. embedding(·) represents the vector representation generated using the BERT model. cos(·) denotes cosine similarity used to measure semantic coherence between sentences. n represents number of sentences already selected in the summary.

Finally, the coverage component (0.15) guarantees that significant query-related information appears in the summary without too constraining the sentence chosen issues. Were calculated by comparing the summary embedding to the document embedding for calculating coverage as described in Eq. (6).

$\operatorname{Coverage}(S, D)=\cos \left(\text {mean}_{\text {embedding}(S)} \text {mean}_{\text {embedding }(D)}\right)$                  (6)

$\text {mean}_{\text {embedding}(S)}=\frac{1}{|S|} \sum_{\text {si } \in S} \text {embedding}(s i)$

$ \text {mean}_{\operatorname{embedding}(D)}=\frac{1}{|\mathrm{D}|} \sum_{\operatorname{di} \in D} \text {embedding}(\mathrm{dj})$

where, S represents the set of sentences selected in the generated summary. D denotes the set of sentences in the source documents. embedding(·) represents the BERT-based sentence embedding representation. $ \text {mean}_{\text {embedding}(S)}$ represents the average embedding vector of all sentences in the generated summary. $ \text {mean}_{\text {embedding}(D)}$ represents the average embedding vector of the source document sentences. cos(·) denotes the cosine similarity function used to measure semantic similarity

These weights are considered a priority, depending on design selection, where relevance and redundancy control are emphasized first, followed by coherence and coverage.

DQN Agent Design: In the DRL model, an agent is built using two deep neural networks, a DQN and a target network with experience replay. The DQN agent is used to learn an optimal policy for selecting sentences, thereby predicting the long-term cumulative reward for each possible action, as illustrated in Eq. (7)

$\text {cumulative reward}=(R t) r_t+\gamma r_{t+1}+\gamma 2 r_{t+2}+\cdots$               (7)

where, $R_t$ denotes cumulative reward starting from time step $t . \,\, r_t$ denotes immediate reward received at time step $t . \,\, \gamma$ denotes discount factor that controls the importance of future rewards $(0 \leq \gamma \leq 1)$.

This cumulative reward reflects the overall quality of the created summary in terms of redundancy reduction, coverage, relevance, and coherence.

The DQN represents a basic decision-making technique in the suggested model, which employs a deep neural network to approximate the Q-value function to generate informative features to represent the states of the RL Bellman model. DQN is used when the action space or states are large (2017). The agent learns which sentences maximize the reward, balancing query relevance and information diversity. The agent utilizes experience replay and policies to balance exploration of new sentence combinations and exploitation of known good policies. At the training step, an agent periodically selects sentences, updates the state, and receives a reward. A target network is periodically synchronized with the DQN network through many episodes and is used to stabilize learning by reducing oscillations in Q-value updates. At each step, experience replay is employed to store past transitions as tuples (state(s), action(a), reward(r), next state(s’), done), which provide samples for training an agent. During the training stage, an agent periodically selects sentences, updates the state, and receives rewards. This allows the model to work in a sequential decision-making process and to generate high-quality extractive summaries. Training the DRL model and exploration: The training approach of the DQN agent proceeds by interacting with the specific environment repeatedly; an EQBS is handled as a sequential decision-making process. At every step, the agent observes the current state vector (a concatenation of the sentence embedding and the query embedding). The DQN architecture consists of fully connected layers with non-linear activations that approximate the optimal action-value function Q(s, a), This results in Q-values representing two possible actions (selecting this sentence or skipping it). It captures a reward based on a multi-component function that integrates BERT-based coherence, MMR for controlling both the relevance and redundancy, and query coverage. These rewards help the agent select sentences that are aligned with the query, contextual, and informative. To guarantee stable learning, a separate target network is preserved and synchronized periodically with the main network. The Q-network is updated iteratively using experience tuples $\left(s, a, r, s^{\prime}\right)$ to enhance convergence stability. These tuples are stored in the replay buffer after each experience and sampled during the training process. An agent, when interacting with an environment during the learning process, balances exploitation and exploration using an ε-greedy strategy, where each chosen sentence is evaluated by the reward function, enabling the agent to progressively move from random exploration to policy-driven action selection based on learned Q-values. Testing stage: This stage of the proposed system involves the implementation of the trained (DQN) agent, forming a final concise extractive summary for each query by concatenating selected relevant sentences tailored to the query. The DQN agent transitions from a learning environment to a completely deterministic decision-making model. Where, after training the agent (in the training stage), input a new corpus of a dataset with the same structure as that used in training the agent (multiple documents, multiple queries, and multiple reference summaries) for an initialization step to apply preprocessing, tokenization, and embedding stages. Unlike training, at this stage, an agent exploits the state space instead of exploring it. The agent is applied only in deduction mode. It selects the action with the highest Q-values acquired during the training stage, guaranteeing a purely exploitative behavior, leveraging the learned policy to assess every sentence corresponding to the query. Moreover, no exploration, no weight updates, and no backpropagation are performed. An environment sequentially provides sentence embeddings concatenated with the query embedding, which was achieved using the BERT model for the trained DQN agent to interact with an environment. At every step, the agent provides Q-values for two possible actions and determines whether to choose or skip a sentence depending on the predicted coherence and relevance. This decision procedure continues until either all sentences are processed or the maximum summary length is reached. The essential components for stabilizing learning during the training stage are the target network and experience replay, which remain inactive during testing. Therefore, the agent is treated as a constant decision model, continuously implementing its understanding of coverage, relevance, and coherence to generate the best extractive query-based summary. Essentially the trained agent assignment is represented as an intelligent selector, autonomously filtering the most informative sentences without any environmental feedback or further adaptability. Summarization-generation stage: This stage involves a trained DRL agent working as a deterministic decision-maker to form a final concise extractive summary by concatenating selected relevant sentences tailored to the query. It assesses every sentence by ensuring that the selected sentences are different from the sentences that were selected before to be included in the generated summary. It captures various aspects of the query without repeating information by utilizing the learned Q-function and greedily choosing just the most relevant sentences. This process will end when the required length of the generated summary is reached. Compared to traditional supervised summarization models, this DQN-based system does not require aligned, labeled data for every possible summary. Instead, it learns from rewards based on its own decisions, making it more flexible and adaptive to unseen queries and document sets. The agent applies a select-or-skip policy sequentially to create an extractive summary.

Table 1. Results of implementing the proposed system compared with the previous

4.png

Figure 4. Tokenization process result

5.png

Figure 5. Sample of episode during training

Table 3. Generated summary compared with reference summary

This research suggested a system for EQBS that integrates DRL with MMR to create diverse, concise, coherent, and relevant summaries from multiple documents in response to user queries. The sentence selection task is formulated as a sequential decision-making issue framed by the MDP. Based on learned policy, an agent selects sentences iteratively. Utilization of MMR in this system assisted in balancing redundancy and query relevance. It helps not only to cover the main topics of the query but also to preserve diversity between chosen sentences. The experimental results explain that the suggested system (DRL + MMR) is more efficient compared with the baseline approaches, such as random selection, Text Rank, and Lex Rank, according to BERT Score, ROUGE-1, ROUGE-2, and ROUGE-L metrics. It is necessary to refer to the proposed system, which carries out a higher semantic similarity to the query and a decreased redundancy rate. Demonstrating the performance of the combination of diversity control through inference and training. Moreover, the results showed that the closely designed reward function, taking into consideration novelty, coverage, relevance, and coherence, was necessary to train an effective and stable agent, although the system achieved a powerful performance. It can be found that a more explicit modeling or direct combination of coherence incentives of phrase transitions might also improve sentence-level coherence. Finally, the suggested DRL+MMR model shows a flexible and strong method for the query-based summarization task, able to learn effective policies for selecting sentences automatically. It successfully handles the main challenges of redundancy reduction, information coverage, summary coherence, and query relevance. Future work: Develop the system to be applied to a huge, diverse document corpus. Enhance the coherence system and apply developed DRL approaches like curriculum learning or transformer-based agents to further enhance the summarization quality. Also, for future work, combination query expansion assesses the DRL agent address the shortest queries effectively. Extending the reward function to contain more criteria.