Electronic Theses and Dissertation

Automatic Speech Recognition (ASR) trilingual adalah sistem yang mentranskripsikan
ucapan tiga bahasa sekaligus. Penelitian ini mengembangkan sistem ASR trilingual
Indonesia-Arab-Inggris melalui fine-tuning model Whisper Small untuk menangani
fenomena code-switching. Whisper Small dipilih karena kemampuan multibahasanya
dari 680.000 jam pre-training. Korpus latih hasil kurasi yang diberi nama INAGRIS
(Indonesia–Arab–Inggris) dikompilasi dari empat belas sumber data publik (antara lain
Common Voice, FLEURS, LibriSpeech, ClArTTS, TITML-IDN), rekaman
code-switching nyata dari podcast Hari Minggoean dan Homostoria serta korpus
QCRI/ESCWA, dan augmentasi code-switching sintetis, dengan total 173,86 jam audio
dan 83.388 utterance. Selain itu, dihimpun langsung himpunan uji independen
INF23_CS berisi 571 ujaran code-switching trilingual nyata dari 57 penutur sebagai
tolok ukur generalisasi. Pelatihan menggunakan strategi curriculum learning dua fase,
yaitu Fase 1 membangun fondasi monolingual (56.010 utterance) dan Fase 2
memperluas ke code-switching (67.167 utterance). Evaluasi pada test set (8.163
utterance) menunjukkan penurunan WER yang signifikan dibandingkan model
zero-shot. WER bahasa Indonesia turun dari 23,93% menjadi 7,78%, bahasa Arab dari
38,05% menjadi 15,82%, bahasa Inggris dari 7,56% menjadi 3,94%, dan segmen
code-switching dari 124,71% menjadi 36,45%. Penurunan serupa terlihat pada dua
metrik pelengkap, dengan MER keseluruhan turun dari 36,50% menjadi 13,80% dan
CER dari 19,67% menjadi 5,79%. Model trilingual terbukti setara dengan tiga model
monolingual khusus, dengan selisih WER di bawah 2 persen pada semua bahasa, dan
mampu mengenali ujaran code-switching. Penelitian ini menghasilkan model ASR
trilingual Indonesia-Arab-Inggris beserta korpus dan tolok ukur evaluasinya.

Trilingual Automatic Speech Recognition (ASR) transcribes speech in three languages simultaneously. This research develops a trilingual Indonesian-Arabic-English ASR system by fine-tuning the Whisper Small model to handle code-switching. Whisper Small was selected for its multilingual capability from 680,000 hours of pre-training. The curated training corpus, named INAGRIS (Indonesian–Arabic–English), was compiled from fourteen publicly available sources (including Common Voice, FLEURS, LibriSpeech, ClArTTS, TITML-IDN), real code-switching recordings from the Hari Minggoean and Homostoria podcasts and the QCRI/ESCWA corpus, and synthetic code-switching augmentation, totaling 173.86 hours of audio and 83,388 utterances. In addition, an independent test set INF23_CS containing 571 genuine trilingual code-switching utterances from 57 speakers was directly collected as a generalization benchmark. Training used a two-phase curriculum learning strategy, where Phase 1 built a monolingual foundation (56,010 utterances) and Phase 2 extended the capability to code-switching (67,167 utterances). Evaluation on a test set of 8,163 utterances shows significant WER reductions compared to the zero-shot model. WER for Indonesian dropped from 23.93% to 7.78%, Arabic from 38.05% to 15.82%, English from 7.56% to 3.94%, and code-switching segments from 124.71% to 36.45%. Similar reductions were observed for the two complementary metrics, with overall MER dropping from 36.50% to 13.80% and CER from 19.67% to 5.79%. The trilingual model proved comparable to three separately trained monolingual models, with WER differences below 2 percent for all languages, and can recognize code-switching speech. This research produces a trilingual Indonesian-Arabic-English ASR model together with its corpus and evaluation benchmark.