Electronic Theses and Dissertation

Teknologi informasi memberi manfaat besar pada kehidupan manusia seperti akses informasi, komunikasi, dan pekerjaan. Salah satu inovasi teknologi informasi adalah sistem pengenalan suara seperti Whisper dari OpenAI, yang mampu melakukan transkripsi dan penerjemahan audio multibahasa serta mengatasi suara bising. Meskipun whisper lebih cenderung dikembangkan pada bahasa Inggris, Whisper memiliki potensi untuk lebih dikembangkan lagi pada bahasa lain seperti bahasa Indonesia. Penelitian ini berfokus pada meningkatkan performa model pre-trained small OpenAI Whisper pada pengenalan suara dalam bahasa Indonesia. Dataset yang digunakan termasuk dataset YouTube pada penelitian sebelumnya yang berjudul "Acoustic Model with Multiple Lexicon Types for Indonesian Speech Recognition" dan Common Voice 9 dari Mozilla Foundation Hugging Face, dengan total 12.243 data pada Common Voice 9 dan 49.022 data pada YouTube. Data tersebut melalui tahapan pre-processing termasuk ekstraksi fitur, tokenisasi dan data collator sebelum digunakan pada model Whisper. Penelitian ini melibatkan fine-tune pre-trained model Whisper small dengan kombinasi hyperparameter batch size dan learning rate pada dataset YouTube dan Common Voice 9. Hasil terbaik dievaluasi dengan mempertimbangkan nilai Word Error Rate. Hasil fine-tune terbaik pada Common Voice 9 diperoleh dengan kombinasi hyperparameter train batch size 8, eval batch size 4, dan learning rate 1E-05, dengan nilai Word Error Rate 12,47. Sedangkan pada dataset YouTube, hasil terbaik dicapai dengan kombinasi hyperparameter train batch size 12, eval batch size 6, dan learning rate 1E-04, dengan nilai Word Error Rate 38,03. Performa dataset YouTube lebih rendah dari Common Voice disebabkan oleh data yang dimiliki dataset YouTube masih belum sepenuhnya bersih, sehingga mempengaruhi hasil dari fine-tuning yang dilakukan.

Kata Kunci: OpenAI, Whisper, Fine-Tune, Transformer, Pytorch, Pre-Trained model, Encoder-Decoder, Pengenalan Suara, Word Error Rate (WER), Hyperparameter

Information technology provides significant benefits to human life, such as information access, communication, and employment. One notabel innovation in information technology is the speech recognition system, exemplified by OpenAI's Whisper, capable of transcribing and translating multilingual audio while mitigating background noise. Despite its initial emphasis on English, Whisper holds the potential for further development in languages like Indonesian. This study centers on enhancing the performance of the small pre-trained OpenAI Whisper model for speech recognition in Indonesian. The utilized datasets encompass the YouTube dataset from the previous study titled "Acoustic Model with Multiple Lexicon Types for Indonesian Speech Recognition" and Common Voice 9 from Mozilla Foundation Hugging Face. These datasets comprise 12,243 instances in Common Voice 9 and 49,022 instances in YouTube. The data undergoes pre-processing stages including feature extraction, tokenization, and data collation prior to its application in the Whisper model. This research involves fine-tuning the small pre-trained Whisper model using combinations of batch size and learning rate hyperparameters on both YouTube and Common Voice 9 datasets. Optimal outcomes are evaluated considering the Word Error Rate. The finest fine-tuning results in Common Voice 9 are achieved with a hyperparameter combination of a train batch size of 8, eval batch size of 4, and learning rate of 1E-05, yielding a Word Error Rate of 12.47. Conversely, on the YouTube dataset, the best results are obtained with a hyperparameter combination of a train batch size of 12, eval batch size of 6, and learning rate of 1E-04, resulting in a Word Error Rate of 38.03. The inferior performance of the YouTube dataset compared to Common Voice can be attributed to the incompletely clean nature of the YouTube data, thereby influencing the outcomes of the conducted fine-tuning. Keywords: OpenAI, Whisper, Fine-Tune, Transformer, PyTorch, Pre-Trained model, Encoder-Decoder, Speech Recognition, Word Error Rate (WER), Hyperparameters