A Systematic Review of Audio-Visual Speech Enhancement: Architectural Evolution, Diffusion-Based Learning, Self-Supervised Models, and Future Perspectives

Audio-Visual Speech Enhancement (AVSE) has emerged as an important research direction for improving speech intelligibility and perceptual quality in acoustically degraded environments through the integration of complementary auditory and visual information sources [1]. Unlike conventional audio-only speech enhancement systems, AVSE frameworks utilize visual cues such as lip movements and facial dynamics to improve robustness under severe noise, speaker interference, reverberation, and signal degradation conditions [2]. The increasing demand for robust multimodal communication systems has accelerated the development of audiovisual speech processing technologies across intelligent communication systems, assistive hearing applications, human–computer interaction, teleconferencing platforms, and real-time multimedia services [3].

Early audiovisual speech processing systems primarily relied on handcrafted feature extraction methods, including Mel-Frequency Cepstral Coefficients (MFCCs), Discrete Cosine Transforms (DCTs), and conventional statistical learning approaches [4]. Although these techniques provided interpretable representations, their performance remained limited under highly variable acoustic conditions, visual occlusions, speaker variability, and unconstrained real-world environments [5]. The emergence of deep learning architectures, particularly Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), enabled improved temporal-spectral representation learning and multimodal fusion capabilities [6]. More recently, transformer-based, diffusion-based, and self-supervised learning (SSL) frameworks have significantly advanced audiovisual speech enhancement by improving contextual modeling, multimodal synchronization, perceptual reconstruction quality, and cross-modal representation learning across complex acoustic scenarios [7].

The rapid evolution of AVSE architectures has been accompanied by substantial growth in audiovisual benchmark datasets. Early datasets such as GRID, LRS2, and LRS3 provided synchronized audiovisual speech corpora for controlled experimental evaluation. However, many currently available audiovisual datasets still suffer from limited vocabulary diversity, environmental variability, and multilingual coverage. To address these limitations, several large-scale datasets have recently been introduced, including AVSpeech, VoxCeleb2-AV, and MuAViC. These datasets include large numbers of speakers, multilingual speech content, spontaneous audiovisual recordings, and diverse recording environments, enabling broader generalization and more realistic robustness evaluation across different scenarios.

Despite these advances, considerable variability still exists among current datasets with respect to data collection procedures, annotation quality, demographic diversity, synchronization quality, and recording environments. Similarly, AVSE methods also vary substantially in terms of architectural design, multimodal fusion strategies, and feature representation techniques. Consequently, benchmarking methodologies and evaluation procedures differ significantly across studies, making fair comparison and reproducibility increasingly difficult.

In parallel with the growth of large-scale audiovisual datasets, AVSE methods have achieved substantial improvements in recent years, particularly through diffusion-based generative frameworks and self-supervised multimodal learning approaches. These methods have demonstrated strong capabilities in multimodal representation learning and perceptual enhancement under severely degraded acoustic conditions. Nevertheless, despite these improvements, current AVSE systems still face several important limitations. Advanced architectures often introduce high computational complexity, increased memory consumption, longer inference latency, expensive training procedures, and reduced deployment efficiency in real-time applications, especially in edge-based environments.

Diffusion-based AVSE architectures, in particular, have shown strong perceptual reconstruction performance in audiovisual enhancement tasks. However, these architectures frequently rely on iterative inference processes that are computationally expensive and difficult to deploy in latency-sensitive applications. Consequently, there is growing interest in developing more efficient and deployment-oriented diffusion-based AVSE frameworks suitable for real-time and resource-constrained environments. Future research is expected to focus on improving the efficiency, scalability, and synchronization stability of diffusion-based AVSE systems.

In addition to perceptual enhancement, increasing attention is being directed toward integrating SSL frameworks such as HuBERT, wav2vec 2.0, and AV-HuBERT with multimodal diffusion models. Such integration may reduce dependence on manually labeled data while improving adaptive contextual representation learning, cross-modal alignment, and robustness under severe acoustic degradation conditions. Despite their potential, diffusion-based AVSE systems still face challenges related to synchronization instability, computational scalability, cross-modal consistency, and deployment feasibility in real-world applications.

Although lightweight AVSE architectures have recently demonstrated promising real-time performance, their enhancement capability often degrades in highly challenging acoustic environments. As a result, achieving a balance between perceptual enhancement quality, synchronization reliability, computational efficiency, and deployment feasibility remains a major challenge in modern AVSE research [8].

Although significant progress has been made in AVSE research using deep learning approaches, many existing studies still exhibit important limitations related to methodology, experimental design, reproducibility, benchmarking consistency, and ethical considerations. These limitations affect the reliability of reported performance results, the fairness of comparisons among different AVSE methods, and the practicality of real-world deployment. Such challenges arise not only from individual studies, but also from variability in multimodal datasets, evaluation protocols, frame synchronization strategies, and the rapid evolution of deep learning architectures and training methodologies.

To address these challenges, it is necessary to evaluate the performance of models trained or optimized on one dataset across multiple AVSE benchmarks rather than relying solely on single-dataset evaluation. Cross-dataset generalization assessment should involve datasets that are as diverse as possible with respect to several factors, including (1) acoustic conditions, (2) language and accent variability, (3) speaker demographics, (4) recording environments, and (5) synchronization quality. This is particularly important for transformer-based, diffusion-based, and self-supervised AVSE models, which are typically trained on large-scale datasets and may inherit biases, synchronization inconsistencies, or dataset-specific characteristics from the training data.

Consequently, modern AVSE review studies must extend beyond providing a simple summary of existing methods and datasets. A comprehensive analytical review should also examine architectural design strategies, multimodal fusion approaches, feature representation methods, benchmarking methodologies, computational efficiency, synchronization reliability, deployment feasibility, multilingual adaptability, demographic fairness, privacy considerations, and reproducibility challenges.

Therefore, this review provides a structured analytical study of AVSE research published between 2020 and 2025 and selected from IEEE, Elsevier, Springer, and ACM databases using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-based study selection procedures. A total of 102 studies were analyzed to investigate transformer-based architectures, diffusion-based frameworks, self-supervised multimodal learning systems, and lightweight deployment-oriented AVSE models. In addition, this review examines multimodal fusion strategies, feature extraction techniques, dataset characteristics, evaluation methodologies, benchmarking practices, and performance-efficiency trade-offs across modern AVSE systems.

Particular attention is given to the relationship between perceptual enhancement quality, synchronization performance, computational complexity, and real-time deployment capability. The review also discusses several emerging research directions, including multilingual low-resource AVSE, privacy-aware multimodal learning, efficient edge-based audiovisual processing, self-supervised representation learning, and large-scale generative multimodal frameworks. The primary outcome of this paper is therefore a structured analytical overview of current AVSE research trends rather than a purely comprehensive summary of the field.

Table 1. Inclusion and exclusion criteria used for study selection

The comparative synthesis indicates that transformer-based and diffusion-based architectures generally achieve improved perceptual enhancement quality and multimodal contextual modeling performance relative to conventional CNN-RNN systems, particularly under highly degraded acoustic conditions [22]. These improvements are primarily associated with enhanced long-range dependency modeling, attention-driven fusion mechanisms, and more effective multimodal representation learning capabilities. However, the increased adoption of transformer-based and diffusion-based architectures is also associated with substantially higher computational complexity, memory consumption, inference latency, and deployment cost, particularly in resource-constrained or real-time operational environments [23].

The other hand, lightweight multimodal architectures for efficient processing have been investigated and compared to larger alternatives. On perceptual processing, better audio-visual synchronization and better scalability are more important. Most approaches that have the potential for efficient processing, low memory demand, and real-time capability achieve higher quality in multimodal input. In contrast, approaches that employ large transformer architectures and diffusion models for unsupervised efficient processing show lower robustness and poor perceptual quality, especially under severe acoustic degradation and larger variations in large multilingual and unconstrained datasets rather than in a laboratory setup. However, the current supervised audio-visual representation learning often achieves better robustness and generalization across domains if trained with balanced datasets, limited multilingual capability, and a lack of demographic diversity. Moreover, very few unsupervised approaches suffer from several restrictions that impede a fair comparison, for example inconsistent benchmarks, and omissions in the development of fairness-oriented benchmarks. Such issues, however, are becoming more important as the AVSE approaches are increasingly deployed in large-scale real-world applications.

Recent, generative-based audio-visual reconstruction techniques However, numerous challenges to current AVSE methodologies and AVSE advancements, are largely rooted in multimodal contextual learning, self-supervised audio-visual representation learning, and potential large-scale multimodal AVSE deployments, remain to be addressed. The most pressing challenges and SE in real-world scenarios.

Table 4. Summary of convergent trends and implications in AVSP evolution

5.1 Evolution of learning paradigms

The majority of studies related to AVSE have moved from feature extraction for both audio and visual streams, and their subsequent combination, to the multimodal learning of acoustic and visual information in context within noisy and partially degraded audio environments. As opposed to the previously handcrafted features that were combined by using earlier AVSE systems, the features are now more often created by using CNNs and RNNs for localized temporal modeling. Most recently, a multitude of approaches has emerged that use transformers for modeling long-range dependencies, as well as diffusion models for modeling complex audio in particular. These approaches additionally include attention for the fusion of the modalities, as well as a number of other aspects related to the multimodal contextualization of synchronized audiovisual streams. Although earlier AVSE systems primarily focused on achieving the highest possible signal reconstruction quality, current approaches pursue a broader set of goals. In addition to higher perceptual quality of restored video, the focus is on improved intelligibility and robustness in a wide variety of acoustic environments. With respect to learning features in an unsupervised fashion, numerous approaches for self-supervised multimodal learning have recently been introduced. They enable efficient learning of features for both audio and visual modalities without the need for large amounts of labeled data for audiovisual pairs. Thanks to their design, which is based on multiple domains and implemented in multiple languages, these approaches are particularly suitable for a wide range of applications and can be flexibly adapted to the requirements of the specific AVSE task at hand. In addition to better contextual modeling, recent research focuses on improving the stability of advanced AVSE systems for real-world applications. While recent approaches clearly excel in terms of their learning capabilities, the requirements on computational resources, training time, and memory for recent approaches such as transformer-based and diffusion-based methods are significantly higher than those for traditional CNN-RNN-based methods. Thus, current research tries to find a suitable balance between a system’s learning capabilities and its scalability, synchronization robustness, and applicability.

5.2 Performance-efficiency balance

The comparative analysis across the reviewed architectural families consistently demonstrates a trade-off between perceptual enhancement performance and computational efficiency. Transformer-based and diffusion-based AVSE systems generally achieve stronger PESQ, SDR, STOI, and WER performance under acoustically degraded conditions due to their superior contextual modeling capability and attention-driven multimodal integration [57, 58]. However, these improvements are frequently accompanied by increased computational complexity, inference latency, memory consumption, and energy requirements.

By contrast, lightweight multimodal frameworks demonstrated stronger deployment feasibility for real-time and resource-constrained environments [28, 59]. Such architectures often employ reduced parameterization, compressed multimodal representations, or computationally efficient fusion mechanisms to support embedded and edge-oriented applications. Although lightweight systems may exhibit lower perceptual reconstruction performance under highly variable acoustic conditions, they remain important for low-latency operational scenarios such as assistive communication systems, mobile devices, and hearing-support technologies [60].

The reviewed literature therefore suggests that future AVSE system design should not prioritize perceptual enhancement quality in isolation. Instead, robust multimodal architectures increasingly require balanced optimization strategies capable of jointly addressing computational scalability, synchronization reliability, energy efficiency, and deployment-oriented adaptability across heterogeneous operational environments.

5.3 Dataset influence and multilingual generalization

An important factor influencing the robustness of AVSE systems, their reproducibility, and their generalization capability to new domains and different domains is the design choice made for the dataset used for training and evaluation. A large number of audio-visual datasets have been recently published, ranging from small, controlled datasets that can be easily synchronized, such as GRID and LRS2, to very large, in-the-wild datasets, which are also used for AV speech recognition, such as AVSpeech and MuAViC.

Large datasets have their own set of challenges, and current methods for handling synchronization instability, metadata of varying quality, imbalanced datasets, noisy labels, and datasets that are not representative of all possible aspects of multimodal human communication all pose challenges for comparative study. Additional challenges include the preprocessing steps that are taken, all of which can negatively impact the reproducibility of results, the train/test split that is not standardized, and the way in which alignment is performed and corrected.

In summary, while increasing dataset diversity is not sufficient to guarantee AVSE robustness yet, more emphasis needs to be placed on transparency of synchronization, on demographic balance, on multilingual coverage, and on standardized evaluation protocols in order to support fairness-aware multimodal evaluation across a variety of deployment scenarios.

5.4 Theoretical and practical integration

A growing trend in current AVSE research is to combine signal-based evaluation metrics with more perceptual and deployment-oriented metrics to assess the quality of AVSE systems in various applications [61-63]. Metrics that measure signal reconstruction quality, such as SDR and PESQ, provide the basic evaluation criteria for perceived speech quality. Other metrics, for example, STOI, WER, and AV-MOS, are used to assess the degree to which a system improves the intelligibility of processed audio while maintaining other important properties of synchronized audio-visual signals.

Recently, there has been a trend in AVSE research toward combining signal processing-oriented evaluation metrics with perceptual and deployment-oriented assessment criteria. While SDR and PESQ must be used for evaluating the signal reconstruction quality and perceptual speech quality, respectively, STOI, WER, and AV-MOS are used for evaluating the other aspects of AVSE systems, such as the intelligibility preservation, multimodal consistency, and recognition-oriented robustness. As shown in Table 4, the evaluation methodologies of recently developed AVSE systems are gradually shifting from isolated metric optimization to using multi-metric assessment frameworks. It is also obvious that no single metric can comprehensively and objectively represent the performance of an AVSE system in various situations. Thus, using integrated and robust evaluation approaches to simultaneously evaluate the perceptual quality, synchronization, implementation complexity, and real-time processing capability of an AVSE system is becoming a necessity for fair benchmarking.

AVSE systems are being explored for various applications in addition to the traditional laboratory-based speech enhancement. These applications include, but are not limited to, assistive communication, hearing aid support systems, telepresence, edge computing based multimodal interfaces, and low resource real time audio-visual communication [34, 60]. The mentioned limitations related to the AVSE deployment, such as latency, power consumption, privacy, and synchronization robustness, are still open issues that need to be addressed in order to make AV.

5.5 Future research and technological outlook

Overall, current AVSE models that incorporate multimodal contextual learning can be combined with self-supervised representation learning and generative audio–visual models for reconstruction. Based on recent trends in AVSE research, several new directions can be anticipated [64]. In particular, future AVSE models could be made more adaptable to multilingual users and diverse linguistic environments. Another strand of emerging research within AVSE focuses on deployment-oriented optimization for various scenarios, exploring a wide range of architectures, diffusion methods, and audio/visual processing techniques to identify the best deployable solution. These methods include, but are not limited to, extremely lightweight architectures, efficient diffusions, adaptive audio/visual compression, and highly efficient edge-based real-time processing using AV inference systems [65].

Third, for now, evaluation protocols for fairness and reproducibility are under-explored. There is a lack of diversity in terms of demographic groups, multilingual representation is not yet consistent, and preprocessing steps as well as synchronization steps are not transparent enough to allow for reliable benchmarking. Future work should focus on creating more fair datasets for AVSE and, in addition to that, reporting results in a more standardized way, making synchronization steps transparent; and, lastly, providing the community with reproducible and viable benchmarking methods.

Finally, and more importantly, as already mentioned in Section 7, explaining and interpreting decisions made by multimodal AVSE systems is a significant open challenge that affects not only its deployment but also a range of other issues, including safety, reliability, and trustworthiness. Visualization of attention, representation space analysis, and, even more importantly, modeling of an interpretable latent space within multimodal SE frameworks offer a promising route for achieving these goals and thus enhance decision-making transparency and empower users of AVSE systems.

In summary, future research should strive to balance improvements in perceptual quality with the integration of several important additional aspects: robustness, scalability, fairness, stable synchronization, efficiency, and practicality for deployment in various multimodal interaction environments.

Table 7. Emerging future research priorities in Audio-Visual Speech Enhancement (AVSE) and their expected methodological impact

7.1 Foundation-scale multimodal models

Looking to scale to process a large amount of information and to jointly process the acoustic and visual information in the different contexts in which they are used will probably be the most relevant for a future application in various contexts [10, 11]. Indeed, although conventional architectures based on CNN-RNN have been able to process the information from the different sensors, more recent multimodal architectures, such as transformer-based, diffusion-based, and self-supervised multimodal architectures, are able to process the information in a more effective way to model the context, especially in cases of degradation of the acoustic information and of occlusion of the visual information [12, 24]. However, although these architectures are able to process the information in a more effective way to model the context, their greater complexity, in terms of the number of parameters and the depth of the architecture, results in greater computational complexity, greater memory consumption of memory and a greater inference latency in studies [67, 68]. Thus, future research should not be limited to the study of the architectures able to process the information in a more effective way to model the context, but it should also focus on the study of the efficient ways to process the information from the different sensors, i.e., on the study of the efficient multimodal fusion mechanisms able to process in real time a large amount of information [82]. In addition to the above-mentioned aspects, the results obtained with the architectures able to reduce the dependency on the task-specific fine-tuning, by means of more general multimodal representation learning, are also very promising for future applications, since, thanks to the learning a more general representation, it is possible to process datasets of different sizes and also to process information in multiple languages, with a reduced number of annotations and a reduced sensitivity to the synchronization between the different modalities [30, 76].

7.2 Hybridization of diffusion and self-supervised learning

Diffusion-based AVSE architectures have generally achieved strong perceptual quality in audiovisual reconstruction. However, these architectures often rely on iterative inference processes that can be computationally expensive, making their deployment in latency-sensitive applications particularly challenging. Consequently, there is growing interest in developing more efficient and deployment-oriented diffusion-based AVSE frameworks suitable for real-time and resource-constrained environments. It is anticipated that future research will focus on improving the efficiency and scalability of diffusion-based architectures for AVSE applications.

In addition to perceptual enhancement, increasing attention is being directed toward integrating SSL frameworks such as HuBERT, wav2vec 2.0, and AV-HuBERT with multimodal diffusion models. Such integration may reduce dependence on manually labeled data while improving adaptive contextual representation learning and cross-modal alignment. Furthermore, conditioning enhancement models on visual contextual embeddings is considered a promising direction for improving AVSE performance under severe acoustic degradation conditions.

Despite their potential, diffusion-based AVSE systems still face several important challenges, including synchronization instability, computational complexity, cross-modal alignment difficulties, and limited scalability in real-time deployment scenarios. Addressing these limitations is expected to play a central role in the future development of robust and efficient AVSE systems.

7.3 Multilingual and culturally inclusive Audio-Visual Speech Processing

The majority of current AVSE research is dependent on English-dominant audio-visual datasets for training, such as LRS2, LRS3, AVSpeech, and the Audio-Visual parts of VoxCeleb2-AV [1, 61]. While the datasets of large-scale multimodal training data are very valuable for training, currently, there is a lack of solid evidence for robust AVSE in multilingual, cross-accent and cross-cultural setups [8, 21]. For future AVSE research, therefore, multilingual evaluation frameworks, low-resource audio and video data, as well as cross-domain evaluation protocols, are required to improve linguistic diversity and to counterbalance demographic imbalances of current multimodal learning systems [77, 80, 82]. Since AV processing deals with speech, which is language-specific and thus changes depending on the speaker’s articulation, i.e. how they produce sounds, and synchronization, i.e. how audio and video are locked in time, which is also language-specific and thus changes across speech communities, it is particularly important to increase linguistic diversity in current AVSE research. In recent years, several approaches for multilingual SSL have been proposed, which aim to improve cross-lingual transferability by learning to represent data in a shared multimodal latent space [63, 76]. However, even for AV processing within a single language, current AVSE is affected by several major issues, i.e. dataset imbalance, annotation inconsistencies, and synchronization variability, which make it difficult to perform comparative evaluations on datasets with different demographics in a fair manner. Future progress in multilingual AVSE therefore not only depends on increasing the size of used datasets, but also on improving transparency of datasets, demographic balance, synchronization as well as reproducibility of benchmarking experiments.

7.4 Ethical, fair, and privacy-preserving Audio-Visual Speech Processing

As systems for AVSE become increasingly important for applications such as assistive communication, telepresence, surveillance, and real-time human-computer interaction, fair evaluation and privacy protection will become more important in future multimodal research [13, 81]. The reviewed studies lack demographic transparency and do not consistently report the fairness-related evaluation procedures that are already used in AVSE. Therefore, future benchmarking approaches should provide more demographic information about the used data and report more consistently on performance, bias and synchronization transparency. Moreover, the approaches should be reproducible [17, 82].

Privacy-aware multimodal learning for AVSE is a growing research area. Audiovisual datasets contain time-synchronized facial and speech signals and thus present critical dimensions of human multimodal signals that require special attention to risks of multimodal data collection and sharing for AVSE learning. In order to reduce risks for misuse or for large-scale centralized repositories of sensitive human multimodal signals, recent multimodal learning approaches employ techniques typical for cross-modal mapping, such as federated multimodal learning, on-device inference, or learning privacy-preserving audio- and visual- representation spaces [71, 72, 82]. In summary, future benchmarking efforts for assessing AVSE performance must combine the already established assessment criteria for performance, bias, and fairness with privacy-sensitive practices for data and datasets for AVSE learning.

7.5 Real-time and edge-deployable multimodal systems

As previously stated in previous studies [42, 71], for future deployment-oriented AVSE systems, in addition to improved perceptual reconstruction quality, it is required to provide real-time processing, high computational efficiency, and low-latency audiovisual inference, suitable for the embedded and edge-oriented environments. A variety of mobile human-computer interfaces (HCl) (MHCls) are currently under development. Such MHCls and other wearable and telepresent HCls, as well as various low-power multimodal human-computer interaction (MMHCl) platforms, require efficient AVSE models, suitable for real-time processing, and running on limited resources. Although model pruning, quantization, knowledge distillation, and, in particular, lightweight multimodal encoders, significantly improve the deployment of the AVSE models, reducing their memory consumption and their respective processing costs, in many cases, decreasing the model’s computational complexity causes a number of trade-offs affecting its perceptual enhancement quality, its ability to ensure synchronization between the reconstructed audio and video streams, and its ability to model various real-world contextual information. Therefore, the focus of future research in the area of AVSE should be placed on developing and evaluating a number of perception-oriented, balanced, and fairly optimized approaches that, simultaneously, improve the quality of the reconstructed audio-visual information, increase the system’s processing speed, improve the processed streams’ synchronization, and ensure highly energy-efficient and efficient AVSE models’ on a variety of highly heterogeneous environments, and highly powerful platforms.

The future of AVSP research is moving in six interrelated directions, which will consolidate speech processing (SP) and make it a fully-fledged area of human-oriented multimodal intelligence. The future of SP will be determined by the interplay of the following six axes: the scalability of new methods, their inclusiveness, their ethics, and their efficiency. In this sense, the future of SP will depend on the unification of the following three areas: diffusion-based learning, fairness-aware governance, and real-time deployment. In this sense, the future of SP will become a technological as well as an ethical benchmark for the development of next-generation AI systems.