Textless Speech-to-Speech Translation from Nepali to English Using Discrete Units: A Deep Learning Approach

Abstract

This study implements direct speech to speech translation eliminating the conventional need for Automatic Speech Recognition (ASR) and Machine Translation (MT) by leveraging HuBERT for discrete unit extraction, a Transformer-based encoder-decoder for translation, and HiFi-GAN for speech reconstruction. Inspired by recent advances in unit-based S2ST, the model is trained on the multilingual FLEURS dataset for Nepali to English language translation. It achieved a BLEU score of 42.80, demonstrating both translation accuracy and natural-sounding speech synthesis. This work offers a scalable and effective solution for low-resource languages like Nepali.