Electronic Theses and Dissertation

Proses penilaian jawaban esai dan jawaban singkat di perguruan tinggi menghadapi tantangan terkait beban kerja dosen dan konsistensi penilaian. Penelitian ini merancang dan membangun sebuah pipeline penilaian otomatis berbasis Retrieval-Augmented Generation (RAG). Sistem ini menggunakan model bahasa besar atau Large Language Model (LLM) berupa model Generative Pre-trained Transformer sumber terbuka (GPT-OSS) 120B dengan 120 miliar parameter sebagai mesin penalaran utama, penelusuran konteks dari basis pengetahuan, pemeringkatan ulang, serta pendekatan fusi bukti ordinal untuk memperbaiki hasil penilaian. Evaluasi utama dilakukan menggunakan benchmark 300 sampel dari dataset Engineering Short Answer Feedback (EngSAF), sedangkan data lokal mata kuliah Jaringan Komputer digunakan sebagai studi kasus awal. Kinerja sistem diukur menggunakan akurasi, Cohen’s Kappa berbobot kuadratik atau Quadratic Weighted Kappa (QWK), Mean Absolute Error (MAE), deferral rate, serta metrik kualitas penelusuran. Hasil terbaik diperoleh oleh fusi bukti ordinal dengan akurasi 0,710, QWK 0,739, MAE 0,303, dan deferral rate 0,003. Prototipe juga berhasil diintegrasikan ke Sistem Pembelajaran Daring Universitas Syiah Kuala (SPADA USK) dan diuji oleh dua dosen melalui User Acceptance Testing (UAT) dan Technology Acceptance Model (TAM), dengan seluruh skenario UAT berhasil dan rata-rata TAM 4,07 dari 5. Hasil ini menunjukkan bahwa RAG membantu penilaian otomatis, dengan hasil paling kuat diperoleh ketika RAG diperkuat dengan contoh jawaban berlabel dan penggabungan beberapa bukti penilaian.

Kata kunci: penilaian otomatis, RAG, Moodle, GPT-OSS 120B, EngSAF

The assessment process for essay and short answer responses in higher education faces challenges regarding lecturer workload and grading consistency. This research designs and develops an automated assessment pipeline based on Retrieval-Augmented Generation (RAG). The system uses a 120-billion-parameter Large Language Model (LLM), the open-weight Generative Pre-trained Transformer model (GPT-OSS 120B), as the main reasoning engine, along with context search from a knowledge base, re-ranking, and ordinal evidence fusion to improve grading results. The main evaluation uses a balanced 300-sample benchmark from the Engineering Short Answer Feedback (EngSAF) dataset, while local Computer Networks course data is used as an initial case study. System performance is measured using accuracy, Quadratic Weighted Kappa (QWK), Mean Absolute Error (MAE), deferral rate, and context search quality metrics. The best result is achieved by ordinal evidence fusion with an accuracy of 0.710, QWK of 0.739, MAE of 0.303, and deferral rate of 0.003. The system was also integrated into Sistem Pembelajaran Daring Universitas Syiah Kuala (SPADA USK) and tested by two lecturers through User Acceptance Testing (UAT) and the Technology Acceptance Model (TAM), with all UAT scenarios completed successfully and an average TAM score of 4.07 out of 5. These results show that RAG supports automated grading, with the strongest result is obtained when RAG is strengthened with labeled answer examples and multiple grading evidence signals. Keywords: automated assessment, RAG, Moodle, GPT-OSS 120B, EngSAF