Evolution of Speaker Recognition: From Handcrafted Signal Processing to Deep Learning-Based Representation Learning

Historically and commercially, voiceprint recognition is also called speaker recognition and is a foremost biometric modality which is based on the speaker's unique and recognizable characteristics for identification and/or authentication. In this review, the term speaker recognition will be used as the main technical term, equivalent to the term voiceprint, which is the biometric term for the same. It is a technology that is different from speech recognition – which aims to capture the spoken word – and particularly appealing for its non-intrusiveness, low cost, and ability to integrate into remote channels, including telephony and internet-based technology. That is the reason it is used in a variety of applications, such as access control in high-security areas, verification of financial transactions, forensic analysis, personalization of consumer electronics, and smart assistants. A remarkable technological evolution is revealed by tracing the field's growth from its humble beginnings in digital signal processing (DSP) to the present day, when artificial intelligence (AI) is built on deep learning architectures.

There are classical approaches that provided the groundwork for automated speaker recognition that involved a two-stage process known as feature extraction and back-end modeling. The first step in the field of feature extraction was the introduction of Mel-Frequency Cepstral Coefficients (MFCCs) [1]. MFCCs are able to give a compact and effective description of the power spectrum of the vocal tract in short durations by transforming the spectrum of the speech signal to the non-linear Mel scale, which mimics the frequency response of the human auditory system. The statistical approaches dominated the modeling phase for more than ten years; in this case, the distribution of the acoustic features of a speaker is modeled as a weighted sum of Gaussian densities, and verification is done by adapting a UBM (trained on a large population of speakers) to a claimant's speech and calculating a log-likelihood ratio. However, this approach, Gaussian Mixture Model-Universal Background Model (GMM-UBM), was very sensitive to session and channel variation [2]. To address this, researchers introduced Total Variability (TV) modeling, which led to the development of the "identity vector (i-vector)" [3]. This paradigm was the high watermark of the classical signal processing era, delivering a strong and compact representation that fuelled much of the progress in this area, and explicitly aiming to eliminate nuisance variability.

However, traditional methods were limited in terms of performance, especially in "in-the-wild" scenarios with high background noise, reverberation, cross-channel effects, and short-duration utterances, due to their dependence on handcrafted features and statistical models that cannot sufficiently represent the high-dimensional, intricate, and non-linear nature of speaker-discriminative information. With the dawn of deep learning, driven by vast datasets and powerful computational resources, a paradigm shift was triggered. One of the most powerful capabilities of Deep Neural Networks (DNNs) was the data-driven and automatic learning of hierarchical and highly discriminative feature representations directly from raw or semi-processed speech signals [4], which meant that no manual feature engineering was required and more robust and abstract representations of the speaker's identity could be identified.

Deep learning was first utilized by employing DNNs as strong classifiers to generate embeddings of speakers. A first example of this was the "d-vector" system, in which the activations from a hidden layer of a DNN were extracted as a fixed-dimensional speaker representation [5], which was trained to classify speakers; subsequently, more elaborate architectures designed specifically for sequence data were developed. The combination of these properties resulted in the development of the "x-vector" architecture, a widely used contemporary baseline [6] due to the ability of Convolutional Neural Networks (CNNs) to capture local spectro-temporal patterns and of Recurrent Neural Networks (RNNs) and their variants, e.g., Long Short-Term Memory (LSTM) units, to model long-range temporal dependencies. X-vectors are generally created by forward-passing the features through the Time Delay Neural Network (TDNN) architecture and a statistical pooling layer, which consolidates frame-level features into a strong utterance-level embedding, especially built to support variable-length inputs.

There are two major trends in the current state of the art: end-to-end (E2E) systems and attention-based architectures. E2E models are trained to directly optimize the final verification metric (e.g., Equal Error Rate (EER)) and, as such, can reduce the pipeline and sometimes improve performance. At the same time, great success has been shown by the transformer architecture with its self-attention mechanism, as the model is able to dynamically weight the importance of each part of an utterance when generating the final speaker embedding. Furthermore, even more contextually rich [7] representations have been produced thanks to the abundance of large-scale, public datasets like VoxCeleb [8] that consist of millions of utterances from thousands of speakers.

Unlike previous surveys that focus primarily on isolated components of voiceprint recognition, a comprehensive paradigm-shifting roadmap is provided in this review. Traditional signal processing is contrasted with deep learning models under a unified taxonomical framework, highlighting the transition from handcrafted features to complex deep embeddings [9]. The theoretical principles and limitations of traditional signal processing methods, such as MFCCs and i-vectors, are first described, before the development of deep learning systems is systematically surveyed, starting from the simple d-vector models to the advanced x-vector and E2E architectures. Each stage is analyzed in comparison with one another to draw out the performance improvements and methodology changes. Finally, a structured review of recurring issues is provided, such as robustness to spoofing attacks, algorithmic fairness and bias, computational efficiency, and emerging future avenues like pre-trained foundation models that are developing the next generation of speaker (voiceprint) recognition technology.

A paradigm shift has occurred in the speaker recognition field through the use of deep learning techniques, whereby the capacity to obtain highly discriminative speaker representations has been greatly increased. Unlike handcrafted features like MFCCs and statistical modeling, the automatic learning of features directly from raw signals or preprocessed audio signals is enabled by deep learning methods. In this section, the development of deep-learning-based speaker recognition systems is discussed, starting from early neural network-based models and the introduction of d-vectors (Section 3.2), followed by the use of explicit models of temporal dynamics such as Convolutional and Recurrent architectures (Section 3.3).

The current baseline x-vector architecture is then discussed (Section 3.4), followed by a description of the design of loss functions that favor highly discriminative embeddings (Section 3.5). An overview of recent developments in speaker representation learning and the foundations of state-of-the-art systems is provided in this review, which are exhaustively covered by these subsections.

3.1 A paradigm shift from handcrafted features to learned representations

The most paradigm-shifting property of deep learning is its ability to learn representations. Optimal and hierarchical representations of features can be learned by DNNs directly from speech data, bypassing the information bottleneck of manual feature engineering, while complex, non-linear, and highly discriminative patterns in the speech signal (which may not be apparent to traditional algorithms [23]) are extracted. Initially, these networks were fed low-level acoustic features such as log Mel-filterbank energies, which are less heavily processed than MFCCs and thus retain more raw information, The deep, multi-layered structure of a neural network allows it to construct a hierarchy of features: simple, local spectro-temporal events (like edges or basic phonetic shapes) may be detected by the initial layers; these simple features are then composed by subsequent layers into more complex and abstract representations (such as formant transitions or syllable-level patterns); and these abstract features are finally mapped into a space that is highly discriminative of speaker identity. This form of automated, data-driven feature learning, optimised concurrently with the classification task, is what makes deep learning-based systems perform so well and much more robust to noise and other acoustic variations, particularly the ability of the data sets to be trained without overfitting, through the use of large-scale, publicly available speech corpora and crucial techniques such as data augmentation [24, 25] (whereby the training set is artificially expanded by adding noise, reverberation or speed/pitch perturbations, significantly enhancing the model's ability to generalise).

3.2 Early architectures: Deep Neural Networks and the genesis of the d-vector

The first successful application of deep learning to the problem of speaker recognition was to repurpose architectures that have already achieved revolutionary results in the closely related problem of automatic speech recognition (ASR) [26]; the first approach was based on using a standard feed-forward DNN trained as a speaker classifier. The input to the network is a feature vector, consisting of features—such as 40-dimensional filterbanks—for a small temporal context window of neighboring frames, fed through multiple hidden layers that consist of non-linear activation functions such as Rectified Linear Unit (ReLU), and then through a Softmax output layer that calculates a probability distribution over a fixed number of speakers that were presented during the network's training.

The crucial innovation was not the classification output itself, but the realization that the activations of one of the final hidden layers could serve as a powerful, fixed-dimensional speaker-discriminative embedding [27]; this embedding was termed the d-vector, The rationale is that in order to correctly classify speakers at the output layer, the network is forced to learn an internal representation within its hidden layers where the feature vectors of different speakers are linearly separable, Therefore, the vector of activations from a bottleneck layer just before the final classifier encapsulates a rich, discriminative summary of the speaker's identity within that short time window. In the variant of this approach used for variable-length test utterances, d-vectors were extracted for each context window in turn, and the utterance-level representation was simply obtained by averaging these vectors over the entire length of the test sentence. The use of a fixed context window limited the network's capacity for modeling long-range temporal dependencies, and the simple averaging of the d-vectors across the entire utterance was a suboptimal pooling strategy, which treated all parts of the utterance as being of equal importance.

3.3 Modeling temporal dynamics: Convolutional and Recurrent architectures

To circumvent the deficiencies of standard DNNs, architectures naturally suited for sequential and structured data were adopted. Speech spectrograms were a natural fit for processing by CNNs, which had been used to reach superhuman performance in image recognition [28, 29]. A 2D spectrogram can be thought of as an image with one axis being the time axis and the other axis being the frequency axis. Filters (kernels) were learned by CNNs over this 2D representation, which allowed them to learn local spectro-temporal characteristics (such as formant trajectories, harmonic structures, and phonetic transitions), making their ability to learn localized, shift-invariant features a more powerful front-end than the fully-connected layers of a standard DNN.

At the same time, RNNs and particularly their gated counterparts, such as the LSTM and the Gated Recurrent Unit (GRU), were investigated as models of long-range temporal dependencies [30]. RNNs have a hidden state which functions as a memory, whereas feed-forward networks do not, making it possible to hold and accumulate information for an entire arbitrary-length utterance. Sophisticated designs are exhibited by the LSTM's input gate, forget gate, and output gate, enabling the model to determine what information to store, forget, or use to generate the current output, thus addressing the vanishing gradient problem that plagued simple RNNs [31]. In the typical LSTM-based speaker recognition system, the final hidden state of the LSTM after processing all the frames of an utterance is taken as the utterance-level speaker embedding, which means that theoretically the final state has captured a summary of the entire utterance. Furthermore, these models were improved by the addition of attention mechanisms, whereby weights are dynamically assigned to different frames during utterance embedding, thereby focusing on the speaker-discriminative part of the speech [32].

3.4 The modern baseline: The x-vector architecture

The x-vector system, typically integrated into the Kaldi ASR toolkit [33], is a carefully designed pipeline that integrates the best of previous systems and is currently the most influential and widely used deep learning-based speaker recognition system in the field today [34], comprising three key components.

The first is a feature extractor, which is usually realized as a TDNN, operating at the frame level. A TDNN is effectively a one-dimensional Convolutional Neural Network (1D-CNN) applied over the temporal axis, where each layer processes a spliced context of frames from the layer below. This structure is computationally efficient and highly effective at learning temporal patterns in speech; the initial layers of the TDNN extract low-level acoustic features, while deeper layers learn more complex and context-aware representations.

The second and most innovative component is the statistics pooling layer. After the frame-level TDNN processing, instead of simple averaging (as in d-vectors) or taking the last state (as in some RNN approaches), the frame-level representations are aggregated over the entire utterance by this layer through computing their first- and second-order statistics: the mean and the standard deviation. The motivation is that the mean vector captures the average acoustic characteristics of the speaker, while the standard deviation vector captures the variability or "spread" of their phonetic realizations, which is itself a powerful speaker-discriminative cue.

The third component is an utterance-level classifier. The concatenated mean and standard deviation vectors are passed through several additional feed-forward layers to produce the final fixed-dimensional embedding, which is the x-vector. The entire network is trained on a multi-class speaker classification task; the extracted x-vector is then typically processed by a backend scoring model, such as PLDA, which has proven to be remarkably effective even when applied to these deep-learned embeddings, whereby system robustness is further enhanced [35, 36].

3.5 Loss functions for highly discriminative embeddings

The choice of loss function used to train the embedding extractor is of paramount importance, as it directly shapes the geometry of the embedding space; the standard approach is to use the Softmax function coupled with cross-entropy loss. For a given training sample x_i with label y_i, the loss is:

${{L}_{Softmax}}=-\left( \frac{1}{N} \right)\underset{i=1}{\overset{N}{\mathop \sum }}\,\log \left[ \frac{\exp \left( W_{\left\{ {{y}_{i}} \right\}}^{T}{{x}_{i}}+{{b}_{\left\{ {{y}_{i}} \right\}}} \right)}{\mathop{\sum }_{j=1}^{C}\text{exp}\left( W_{j}^{T}{{x}_{i}}+{{b}_{j}} \right)} \right]$          (3)

where, x_i is the deep embedding, and W_j and b_j are the weights and bias for the j-th class in the final classification layer as in Eq. (3). While this loss function effectively enforces separability (i.e., it pushes embeddings from different classes apart), it does not explicitly encourage two crucial properties for open-set speaker verification: intra-class compactness (forcing embeddings from the same speaker to be tightly clustered) and inter-class margin (enforcing a larger separation between different speaker clusters).

To address this, a new family of margin-based Softmax losses was developed, which has become the state-of-the-art [37]. These losses work by reformulating the logit $W_{j}^{T}{{x}_{i}}$ in terms of cosine similarity, ||W_j|| ||x_i|| cos(θ_j), where $\theta_j$ is the angle between the embedding x_i and the weight vector for class j. By fixing ||W_j|| = 1 and normalizing the embedding ||x_i|| = s. Consequently, the optimization process shifts toward regulating angular distances on a unit hypersphere. A prominent loss function employing this formulation is the Additive Angular Margin Softmax (AAM-Softmax), widely known as ArcFace [38], which introduces an explicit additive margin in the angular space to enforce tight intra-class compactness and maximize inter-class separability [38]. It introduces an AAM, m, directly into the angle of the target class as in Eq. (4):

$\begin{gathered}L_{A A M}=-\left(\frac{1}{N}\right) \sum_{i=1}^N \log \left[\frac{\exp \left(s \cos \left(\theta_{\left\{y_i\right\}}+m\right)\right)}{\exp \left(s \cos \left(\theta_{\left\{y_i\right\}}+m\right)\right)+\sum_{j=1, j \neq y_i}^C \exp \left(s \cos \left(\theta_j\right)\right)}\right]\end{gathered}$            (4)

This modification makes the optimization problem more difficult, forcing the network to learn features that are not only separable but also highly compact and widely separated in the angular domain; this results in embeddings with significantly improved discriminative power. Other popular variants include CosFace [39], which adds a margin to the cosine value, and SphereFace [40], by applying a multiplicative angular margin. Ultimately, adopting these sophisticated margin-based loss functions has been instrumental in driving state-of-the-art performance in deep speaker recognition architectures [41-44].

As summarized in Table 2, the development of DNN architectures addresses different trade-offs between computational complexity and temporal modeling. While d-vectors paved the way, their reliance on simple frame averaging lost critical dynamic sequence details. CNNs and RNNs introduced localized and sequential modeling, respectively, but were surpassed by TDNN-based x-vectors. The x-vector's integration of a statistics pooling layer (mean and standard deviation) successfully captures both the distribution and variance of speaker characteristics across variable-length utterances.

Table 2 shows an overview of the deep learning architectures that have been used for speaker embedding extraction, the pooling steps used, their key strengths, and their primary weaknesses. This overview places the design decisions made in the highly discriminative embeddings and the subsequent use of margin-based loss functions for open-set speaker verification in context.

In summary, deep learning resolved the limitations of handcrafted feature engineering by enabling automatic, hierarchical representation learning. The evolution from basic d-vectors to statistical-pooling-based x-vectors significantly improved temporal context modeling. When integrated with modern margin-based loss functions such as AAM-Softmax, these models compress intra-class variability while expanding inter-class separation, establishing highly robust baselines for open-set speaker verification.

Table 2. Comparison of deep learning architectures and pooling methods for speaker embedding extraction

Note: DNN = Deep Neural Network; CNN = Convolutional Neural Network; RNN = Recurrent Neural Network; LSTM = Long Short-Term Memory; TDNN = Time Delay Neural Network, GRU = Gated Recurrent Unit, PLDA = Probabilistic Linear Discriminant Analysis.

While the strides made with deep learning are impressive, there are still some key issues that are at the forefront of speaker recognition research. Looking ahead, the field is increasingly concerned not only with accuracy, but also with the security, fairness, and transparency of these powerful biometric systems.

5.1 System robustness and anti-spoofing

Deep models work well when it comes to massive amounts of in-domain data, but are less effective when data is mismatched. There are additional issues that are still under investigation to improve robustness to real-world variability, like short-duration utterances, far-field audio with heavy reverberation, and unseen device channels. But a far more troubling and insidious problem is spoofing attacks – those that try to trick a biometric system with fake voice signals. These attacks are generally considered presentation attacks, and they can be classified as follows:

•Replay Attacks: Replaying a recording of a real user's voice.

•Text-to-Speech (TTS) Synthesis: Employing a synthesis engine that produces speech output that is similar to that of a target speaker's voice.

•Voice Conversion (VC): Alteration of the voice of a source speaker so as to sound like a target speaker.

The newness and advancement of such generative models create a significant security threat; thus, the concept of anti-spoofing, commonly referred to as presentation attack detection (PAD), has been introduced into the scene. A countermeasure (CM) system is a binary classifier that can classify genuine (bona fide) speech versus spoofed speech. Initial methods considered the speaker verification module and the CM module as independent, and their scores were combined at the end. Such multi-task training can yield more secure and integrated systems; however, a tough race is currently developing between the continuous progress of speech synthesis and replay technology and the development of robust CMs.

5.2 Fairness, algorithmic bias, and explainability

Speaker Recognition systems are increasingly used in critical applications such as financial services and access control, leading to significant ethical concerns. One of the concerns is algorithmic bias. Because they rely on big data, deep learning models can learn and magnify biases in the data they receive. A system that is demographically imbalanced due to an uneven number of speakers in a training corpus (e.g., more male speakers than female speakers, or more speakers of one accent than another) will also be an unethical and dysfunctional one, because the system may have a high error rate on the underrepresented group (e.g., high error rate on female speakers, or high error rate on speakers of a different accent) and thus be unsuitable for that part of its user base. Research on this topic is primarily centered on curating more balanced datasets, creating data augmentation and re-weighting techniques to reduce bias, and designing optimization algorithms that are fair to the users.

A similar challenge is explainable artificial intelligence (XAI). Speaker recognition systems of today are indeed 'black boxes'. They generate a score, but they cannot explain why a certain 'decision' has been made; therefore, this lack of transparency is a big hurdle when it comes to user trust, system debugging, and regulatory compliance. Research in XAI for speaker recognition seeks to create methods for understanding the speaker recognition model's reasoning, including the creation of saliency maps that highlight those parts of the spectrogram that were most important for a decision to be made. One of the most critical areas for voice biometric systems in the future is ensuring that they are accurate, secure, fair, and transparent.

5.3 Emerging paradigms: Pre-trained speech models, multimodality, and edge deployment

Beyond the traditional deep learning pipelines, several modern paradigms are actively shaping the future of speaker recognition. First, self-supervised learning (SSL) via large pre-trained speech foundation models (such as wav2vec 2.0 and HuBERT) has emerged as a dominant force. These models leverage hundreds of thousands of hours of unlabeled audio to extract general speech representations, which can then be fine-tuned with minimal target speaker data. Second, multimodal biometric systems are gaining attention by fusing voiceprint recognition with facial and lip-movement features to guarantee security in highly noisy environments. Lastly, to support the deployment of state-of-the-art architectures on mobile and IoT devices, research into lightweight deployment techniques—such as knowledge distillation, network pruning, and float quantization—has become essential to optimize inference speed and energy efficiency without sacrificing performance.