Multi-Model Telugu Speech Recognition: Improving ASR with Dialect Classification and Optimization Techniques

1.1 Speech recognition and Telugu dialects

Speech recognition technology has revolutionized human-computer interaction, enabling natural communication through voice commands. Automatic Speech Recognition (ASR) plays a crucial role in various domains, including virtual assistants, transcription services, accessibility tools, and language learning applications [1]. ASR systems have achieved remarkable success in high-resource languages such as English, French, and Chinese, primarily due to the availability of large annotated speech datasets and extensive computational resources [2]. However, many low-resource languages, including Telugu, face significant challenges due to limited data availability, phonetic complexity, and dialectal variations [3].

1.2 Challenges in Telugu speech recognition

Telugu, a Dravidian language spoken by over 80 million people, is widely used in Telangana and Andhra Pradesh [4]. The challenge arises from phonetic complexity, dialectal variations, and a lack of annotated datasets. Telugu comprises three primary dialects:

(1) Telangana Dialect – Spoken in 33 districts of Telangana, characterized by distinct intonation patterns, phoneme reductions, and nasalization.

(2) Andhra Dialect – Spoken in 9 districts of Coastal Andhra, differing in vowel elongation and phoneme stress.

(3) Rayalaseema Dialect – Spoken in 4 districts (Chittoor, Anantapur, Kurnool, Kadapa), with consonant shifts and unique vocabulary [5].

These variations cause ASR systems to struggle with transcriptions, as the same word can be pronounced differently across dialects. For instance:

Telangana: "నేను వెళ్లాను"

Andhra: "నేను వెళ్ళాను"

Rayalaseema: "నేనె వెళ్ళాను"

1.3 Challenges in Telugu ASR and dialect identification

1.3.1 Limited generalization of traditional ASR models

Traditional ASR models trained on standard Telugu datasets often fail to generalize well across dialectal variations. This is because these models primarily learn from a homogeneous dataset, leading to bias towards mainstream Telugu while failing to accurately transcribe dialectal differences. The existing ASR systems struggle to recognize variations in pronunciation, lexical differences, and phonetic shifts specific to Telangana, Andhra, and Rayalaseema dialects [6]. These dialects exhibit distinct intonation patterns, vowel length variations, and regional vocabulary, which traditional ASR models, such as Hidden Markov Models (HMMs) and hybrid Deep Neural Networks (DNNs), fail to capture effectively.

1.3.2 Lack of large annotated datasets

Unlike English and Hindi, which have extensive annotated speech datasets for ASR training, Telugu suffers from a lack of large-scale dialect-labeled speech datasets. Most publicly available datasets, such as Mozilla Common Voice Telugu, contain standard Telugu speech data but lack dialect-specific labeling [7]. This scarcity of labeled data poses significant challenges for supervised learning-based ASR models. Without sufficient dialect-annotated training samples, models struggle to distinguish regional variations, leading to poor generalization in real-world applications.

To address this, researchers have recently explored advancements in self-supervised learning (SSL). Self-supervised models, such as Wav2Vec2, HuBERT, and Whisper, leverage large amounts of unlabeled speech data and learn robust speech representations without requiring extensive manual annotations [8]. These models fine-tune their feature extraction layers to capture phonetic variations across dialects, compensating for the lack of labeled datasets to some extent.

1.3.3 Phonetic complexity of Telugu

The Telugu phonetic system consists of 56 letters (18 vowels and 38 consonants), including unique retroflex consonants, long vowels, and aspirated sounds, which make phoneme recognition highly challenging [9]. The presence of context-dependent pronunciation changes further complicates ASR performance. Certain letters and syllables undergo morphophonemic alternations, meaning their pronunciation shifts based on preceding or succeeding phonemes, speaker accent, and regional influence.

For instance:

• The pronunciation of "చ" ("cha") may vary subtly between Andhra and Telangana dialects due to vowel length differences.

• Certain nasalized vowels and geminated consonants appear in the Rayalaseema dialect but are absent in other variants.

Traditional ASR models trained on fixed phoneme dictionaries often struggle with such variations. Deep learning-based models, particularly Transformer-based architectures like Whisper and BERT, are better suited to handle phonetic ambiguity by learning contextual dependencies in speech transcriptions.

1.3.4 Real-world noisy environments

In practical applications, ASR models must function effectively in noisy environments with background noise, reverberation, and overlapping speech [10]. Traditional ASR models, which rely on Mel Frequency Cepstral Coefficients (MFCCs) and HMM-based speech recognition, struggle with noise robustness and often misinterpret words when speech is degraded.

Deep learning-based ASR models, such as Wav2Vec2 and HuBERT, improve noise robustness by learning context-aware acoustic representations. These models use self-attention mechanisms and CNN-based feature extractors to filter out irrelevant noise components while preserving meaningful speech patterns [10].

The probability of generating a transcription given an input speech signal can be formulated as

$\mathrm{P}(\mathrm{Y} \mid \mathrm{X})=\prod_{\mathrm{t}=1}^{\mathrm{T}} \mathrm{P}\left(\mathrm{y}_{\mathrm{t}} \mid \mathrm{X}, \mathrm{y}_{1: \mathrm{t}-1}\right)$                (1)

where X represents the raw speech signal and Y the corresponding transcription sequence. The Data augmentation techniques, such as spectrogram masking, adding synthetic noise, and time-stretching, further enhance ASR models’ ability to operate in real-world noisy environments. The Empirical testing under 0–20 dB SNR conditions showed that Wav2Vec2 + BERT maintained 91.7% accuracy, confirming strong noise robustness. This validates the proposed model’s robustness under real-world acoustic interference.

The impact of ASR using the proposed solution is illustrated in Table 1.

Table 1. Impact of ASR with proposed solution

By addressing these challenges, our proposed dialect-aware ASR framework significantly improves Telugu ASR performance, particularly in dialect-rich speech recognition tasks.

1.4 Novelty and synergy among Whisper, Wav2Vec2, HuBERT, and BERT

The proposed framework introduces a hybrid architecture that integrates Whisper, Wav2Vec2, and HuBERT models for ASR, combined with a BERT-based dialect classifier. The ensemble fusion mechanism leverages the robustness of Whisper for noisy speech, the fine-grained acoustic feature learning of Wav2Vec2, and the contextual representation capabilities of HuBERT. Their outputs are weighted and combined to produce optimized transcriptions, which are subsequently processed by BERT for dialect identification. This synergy between multiple ASR and NLP models represents a novel contribution in Telugu speech processing. This hybrid synergy ensures both acoustic robustness (from ASR) and linguistic discrimination (from BERT), leading to a 3–5% improvement in F1-score compared to any single model.

3.1 Dataset explanation

3.1.1 Dataset pre-processing

Before training ASR models and dialect classification models, the raw dataset needs to be cleaned, processed, and structured correctly. The goal is to ensure that the data is high-quality, noise-free, and suitable for training. The dataset was balanced across the three major dialects of Telugu, i.e., Telangana, Andhra, and Rayalaseema, each constituting approximately one-third of the total samples. Label validation was carried out by three native linguists, achieving an inter-annotator agreement (Cohen’s κ = 0.89). Stratified sampling ensured a fair representation of gender, age, and environmental diversity.

3.2 Steps in dataset preprocessing

Step 1: Audio Data Collection

The Telugu dialect dataset was recorded from diverse real-world environments (colleges, offices, parks, roadside) to capture various acoustic conditions. The Mozilla Common Voice dataset was also incorporated to enhance robustness. Speakers of different age groups, genders, and educational backgrounds were included, resulting in 7 hours and 5 minutes of speech covering Telangana, Andhra, and Rayalaseema dialects.

Step 2: Audio Cleaning and Preprocessing

To enhance dataset quality for ASR and dialect classification, all audio files were standardized to 16kHz, mono-channel WAV format. Noise reduction using spectral subtraction and adaptive filtering improved speech clarity, while volume normalization ensured consistent loudness. Silence removal eliminated unnecessary pauses, optimizing efficiency. Finally, segmentation splits long recordings into 3–10 second clips, aligning with ASR training needs and improving transcription accuracy. The complete dataset description is given in Table 3.

Table 3. Dataset description

The relationship between dataset size N and model accuracy can be approximated as

Accuracy $=A_{\infty}-\frac{B}{N}$                 (2)

$A_{\infty}$ is the asymptotic accuracy and $B$ is the dataset efficiency factor.

3.3 Proposed models

3.3.1 Whisper

Whisper, developed by OpenAI, is a powerful end-to-end speech recognition model trained on a large multilingual and multitask dataset. It employs a transformer-based encoder-decoder architecture, where the encoder converts raw speech into log-Mel spectrograms, and the decoder generates text transcriptions. One of Whisper’s key strengths is its robustness to diverse accents, background noise, and different speaking styles, making it highly effective for real-world speech recognition. Unlike traditional ASR models that rely on phoneme-based training, Whisper learns directly from large-scale audio-text pairs, allowing it to generalize well across various speech conditions. However, its autoregressive decoding mechanism makes it computationally expensive, requiring high-end GPUs for real-time applications.

3.3.2 Wav2Vec2

Wav2Vec2, introduced by Meta (Facebook AI), is a self-supervised ASR model designed to learn speech representations directly from raw waveforms. It eliminates the need for manual phoneme labeling by leveraging contrastive learning, where the model predicts masked portions of speech from surrounding audio. The architecture consists of a convolutional feature extractor followed by a transformer encoder, enabling it to capture both local and global speech patterns effectively. Wav2Vec2 is fine-tuned using Connectionist Temporal Classification (CTC) loss, allowing it to directly output text transcriptions without an explicit language model. The CTC loss is defined as

$\left(P(Y \mid X)=\sum_{A \in \operatorname{Align}(X, Y)} P(A \mid X)\right)$               (3)

where, $Y$ is the target text, $X$ is the input audio, and $A$ represents all possible alignments.

This model is particularly advantageous for low-resource languages like Telugu, as it can achieve high accuracy with limited labeled data. Additionally, its non-autoregressive decoding makes it computationally efficient and suitable for real-time applications.

3.3.3 Hidden-Unit BERT (HuBERT)

Hidden-Unit BERT (HuBERT) is another self-supervised ASR model that improves upon Wav2Vec2 by incorporating a masked speech prediction strategy. Inspired by BERT’s masked language modeling approach, HuBERT learns speech representations by predicting masked segments of an audio signal using hidden-unit assignments. The model undergoes two-stage training: first, it learns a coarse representation of speech, and then it refines its understanding through SSL. This hierarchical approach enhances its ability to recognize phonetic and linguistic patterns, making it particularly effective for distinguishing dialectal variations. While HuBERT outperforms Wav2Vec2 in terms of phoneme recognition and generalization, it is computationally more demanding and requires a larger dataset for optimal performance.

3.3.4 Bidirectional Encoder Representations from Transformers (BERT)

Bidirectional Encoder Representations from Transformers (BERT), developed by Google AI, is a transformer-based model widely used for natural language processing tasks, including text classification. In the context of Telugu dialect identification, BERT processes ASR-generated transcriptions to analyze linguistic, phonetic, and syntactic variations across dialects. Unlike traditional NLP models that process text sequentially, BERT employs a bidirectional self-attention mechanism, allowing it to capture contextual dependencies from both past and future words. It is pre-trained using masked language modeling and next-sentence prediction, making it highly effective in understanding subtle differences in dialectal speech patterns. However, its classification performance heavily depends on the accuracy of ASR transcriptions, and its computational complexity requires optimization for real-time deployment.

3.4 Proposed methodology

This section presents an end-to-end ASR and dialect classification system using state-of-the-art deep learning models. The system consists of two key components:

3.4.1 Speech-to-text conversion

Speech-to-text conversion using ASR models and dialect classification using a BERT-based model. For speech recognition, state-of-the-art self-supervised ASR models—Whisper, Wav2Vec2, and HuBERT—are employed to convert raw .wav audio files into text transcriptions without requiring manual labeling. These models leverage SSL techniques to learn speech representations directly from raw waveforms, making them highly effective in handling diverse acoustic conditions and speaker variations.

3.4.2 Dialect classification (BERT-based model)

Once the transcriptions are generated, the dialect classification component utilizes a BERT-based model to analyze the linguistic patterns and classify the speech into one of the three Telugu dialects, i.e. Telangana, Andhra, or Rayalaseema.

Given a transcribed text T, the probability of its dialect classification C follows Bayes' Theorem

$P(C \mid T)=\frac{P(T \mid C) P(C)}{P(T)}$                 (4)

where, $P(T \mid C)$ is the likelihood of text occurring in a specific dialect, and $P(C)$ represents the dialect's prior probability. BERT's bidirectional transformer architecture enables it to capture phonetic and syntactic variations in the transcriptions, ensuring accurate dialect identification. Our approach uses SSL to overcome low-resource limitations. It significantly improves dialect-aware ASR performance, making the system adaptable to real-world speech applications. The BERT model captures dialectal differences by encoding contextual embeddings that reflect phonetic and syntactic variations. For instance, suffix usage ("-ra", "-lu") and unique lexical forms of Telangana are learned as attention-weighted tokens. Attention heatmaps confirmed distinct focus patterns correlating with dialect-specific words, supporting BERT's linguistic interpretability.

3.4.3 Training and testing phases in the ASR system

The training phase is the foundational step in building an ASR system. It begins with data collection, where large-scale speech datasets such as LibriSpeech and Common Voice are gathered. The raw speech data undergoes preprocessing, including noise reduction, normalization, and augmentation, to improve model robustness. Next, feature extraction is performed using Mel spectrograms or MFCCs, converting raw waveforms into numerical representations. The extracted features are then fed into deep learning models like Wav2Vec2, Whisper, and HuBERT, which learn speech patterns through supervised learning. To further enhance language understanding, BERT-based post-processing is applied for grammatical correction and contextual refinement. Finally, optimization techniques, including the AdamW optimizer, dropout regularization, and hyperparameter tuning, help improve the model’s performance and prevent overfitting. Once training is complete, the model is ready for evaluation in the testing phase. The proposed training phase is shown in Figure 1.

Figure 1. Training phase of the proposed model

The testing phase ensures the trained ASR model performs well on unseen speech data. The trained model is loaded, and new speech input is processed through the same feature extraction steps as in training. The ASR model generates a transcription, which is refined by BERT post-processing to improve accuracy. The transcription is then evaluated using key performance metrics such as WER, F1-score, Precision, Recall, and Latency Analysis. If the performance does not meet expectations, the model is sent back for retraining with further adjustments in hyperparameters or additional dataset augmentation.

Figure 2. Testing phase of proposed system

If the model meets the accuracy and efficiency benchmarks, it is deployed for real-world applications with better Accuracy Transcripted data.

This iterative training and testing process ensures that the ASR system is optimized for both accuracy and real-time usability before deployment. The proposed testing phase is shown in Figure 2.

3.4.4 Structured comparison of ASR models

Table 4 presents the comparison of ASR models like Whisper, Wav2Vec2, and HuBERT.

Table 4. Comparison of ASR models: Whisper, Wav2Vec2, and HuBERT

3.5 Optimization methods

AdamW optimization strategy: The learning rate follows a cosine decay schedule, given by

$\eta_t=\eta_{\min }+\frac{1}{2}\left(\eta_{\max }-\eta_{\min }\right)\left(1+\cos \left(\frac{\pi t}{T}\right)\right)$            (5)

where, $\eta_{\mathrm{t}}$ is the learning rate at training step $t$, and $T$ represents the total number of training iterations.

To prevent gradient explosion, gradient clipping is applied as follows

$g_{\text {clipped }}=g_t \cdot \frac{\tau}{\max \left(\tau,\left|g_t\right|\right)}$              (6a)

$\theta_t=\theta_{t-1}-\eta_t g_{\text {clipped }}$                  (6b)

where, $g_t$ is the gradient, and $\tau$ is the predefined clipping threshold.

3.5.1 Algorithm for optimized Telugu ASR model

ASR models (Whisper, Wav2Vec2, HuBERT) with BERT-based dialect classification.

(1) Feature Extraction

Compute feature representations from speech signals:

• Log-Mel Spectrogram (Whisper):

$S_{m e l}=\log$ MelFilterBank $\left(\left|F\left(x_t\right)\right|^2\right)$               (7)

where, $x_t$ is the input waveform, $F$ is the Short-Time Fourier Transform (STFT), and MelFilterBank applies the mel-scale transformation.

• Waveform Embeddings (Wav2Vec2, HuBERT):

$E_{\text {wav }}=f_{\mathrm{Wav} 2 \mathrm{Vec} 2 / \mathrm{HuBERT}}\left(x_t\right)$               (8)

where, $f_{\text {Wav2Vec2/HuBERT}}$ represents the feature extraction model.

(2) Encoder Processing

Apply a Transformer encoder to process extracted features:

$\mathrm{Z}=$ TransformerEncoder $\left(S_{\text {mel }}\right.$ or $\left.E_{\text {wav }}\right)$                (9)

where, Z represents the contextualized speech embeddings.

(3) Decoding (Speech-to-Text Conversion)

Convert speech representations into Telugu text using:

• Whisper (Autoregressive Decoder):

$P(Y \mid Z)=\prod_{t=1}^T P\left(y_t \mid y_{<t}, Z ; \theta\right)$                (10)

where, Y =(y₁, y₂, …, y_T) is the output token sequence and θ represents model parameters.

• Wav2Vec2 (CTC Loss):

$L_{C T C}=-\sum_{(X, Y)} \log P_{C T C}(Y \mid Z)$                   (11)

where, $P_{C T C}(Y \mid Z)$ is the probability distribution over possible label alignments where $X, Y, Z$ represents input, label, and model output respectively.

(4) Text Tokenization

Convert transcriptions into sub word tokens using a tokenizer T

T=Tokenizer (Y)              (12)

where, T=(t₁, t₂, …, t_N) are subword tokens.

(5) Feature Embedding and Dialect Classification by BERT

• Compute BERT embeddings for subword tokens:

E_BERT=f_BERT(T)                (13)

Apply multi-head attention:

$H=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$               (14)

where, $Q, K, V$ are query, key, and value matrices from BERT embeddings, and $d_k$ is the embedding dimension.

• Dialect Classification using Softmax:

$P(C \mid H)=\operatorname{softmax}(W H+b)$               (15)

where, $P(C \mid H)$ is the probability distribution over dialect classes.