Analyzing PEGASUS Model Performance with ROUGE on Indonesian News Summarization

Authors

  • Fatih Fauzan Kartamanah UIN SGD Bandung
  • Aldy Rialdy Atmadja UIN Sunan Gunung Djati Bandung
  • Ichsan Budiman UIN Sunan Gunung Djati Bandung

DOI:

10.33395/sinkron.v9i1.14303

Keywords:

Abstractive summarization, Indonesian language, Natural language processing, PEGASUS, ROUGE

Abstract

Text summarization technology has been rapidly advancing, playing a vital role in improving information accessibility and reducing reading time within Natural Language Processing (NLP) research. There are two primary approaches to text summarization: extractive and abstractive. Extractive methods focus on selecting key sentences or phrases directly from the source text, while abstractive summarization generates new sentences that capture the essence of the content. Abstractive summarization, although more flexible, poses greater challenges in maintaining coherence and contextual relevance due to its complexity. This study aims to enhance automated abstractive summarization for Indonesian-language online news articles by employing the PEGASUS (Pre-training with Extracted Gap-sentences Sequences for Abstractive Summarization) model, which leverages an encoder-decoder architecture optimized for summarization tasks. The dataset utilized consists of 193,883 articles from Liputan6, a prominent Indonesian news platform. The model was fine-tuned and evaluated using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric, focusing on F-1 scores for ROUGE-1, ROUGE-2, and ROUGE-L. The results demonstrated the model's ability to generate coherent and informative summaries, achieving ROUGE-1, ROUGE-2, and ROUGE-L scores of 0.439, 0.183, and 0.406, respectively. These findings underscore the potential of the PEGASUS model in addressing the challenges of abstractive summarization for low-resource languages like Indonesian language, offering a significant contribution to summarization quality for online news content.

GS Cited Analysis

Downloads

Download data is not yet available.

References

A. Joshi, E. fidalgo, E. alegre, & L. Fernández-Robles. (2019). SummCoder: An Unsupervised Extractive Text Summarization Framework Based On Deep Auto-encoder. Expert Syst Appl, 129.

Agus Tri Haryanto. (2024, February 7). APJII Jumlah Pengguna Internet Indonesia Tembus 221 Juta Orang. Asosiasi Penyelenggara Jasa Internet Indonesia (APJII). https://apjii.or.id/berita/d/apjii-jumlah-pengguna-internet-indonesia-tembus-221-juta-orang#:~:text=Asosiasi%20Penyelenggara%20Jasa%20Internet%20Indonesia%20(APJII)%20mengumumkan%20jumlah%20pengguna%20internet,jiwa%20penduduk%20Indonesia%20tahun%202023

Alrasheedi, F., Zhong, X., & Huang, P. C. (2023). Padding Module: Learning the Padding in Deep Neural Networks. IEEE Access, 11, 7348–7357. https://doi.org/10.1109/ACCESS.2023.3238315

Annisa Lutfin, N., Sartika, D., Aisyah Humairah, N., Nur Tuada, R., Rahmadhani, A., & Arifin, S. (2023). PERANAN POJOK BACA DALAM MENUMBUHKAN BUDAYA LITERASI SISWA DI SMPN SATAP LENGGO. 2(4), 757–763. https://etdci.org/journal/patikala/

Aurelia, M., Monica, S., & Girsang, A. S. (2024). Transformer-based abstractive indonesian text summarization. International Journal of Informatics and Communication Technology (IJ-ICT), 13(3), 388. https://doi.org/10.11591/ijict.v13i3.pp388-399

Barbella, M., Risi, M., & Tortora, G. (2021). A comparison of methods for the evaluation of text summarization techniques. Proceedings of the 10th International Conference on Data Science, Technology and Applications, DATA 2021, 200–207. https://doi.org/10.5220/0010523002000207

Barbella, M., & Tortora, G. (2022). ROUGE metric evaluation for Text Summarization techniques. https://ssrn.com/abstract=4120317

Fiok, K., Karwowski, W., Gutierrez, E., Davahli, M. R., Wilamowski, M., & Ahram, T. (2021). Revisiting text guide, a truncation method for long text classification. Applied Sciences (Switzerland), 11(18). https://doi.org/10.3390/app11188554

Hartawan, G., Sa’adillah Maylawati, D., & Uriawan, W. (2024). BIDIRECTIONAL AND AUTO-REGRESSIVE TRANSFORMER (BART) FOR INDONESIAN ABSTRACTIVE TEXT SUMMARIZATION. JIP (Jurnal Informatika Polinema), 10(4), 535–541.

Kirmani, M., Kaur, G., & Mohd, M. (2024). Analysis of Abstractive and Extractive Summarization Methods. International Journal of Emerging Technologies in Learning (IJET), 19(01), 86–96. https://doi.org/10.3991/ijet.v19i01.46079

Koto, F., Lau, J. H., & Baldwin, T. (2020). Liputan6: A Large-scale Indonesian Dataset for Text Summarization. http://arxiv.org/abs/2011.00679

Kudo, T., & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 66–71. http://www.statmt.org/moses/

Laksana, M., Karyawati, A., Putri, L., Santiyasa, I., Sanjaya ER, N., & Kadnyanan, I. (2022). Text Summarization terhadap Berita Bahasa Indonesia menggunakan Dual Encoding. JELIKU (Jurnal Elektronik Ilmu Komputer Udayana), 11(2), 339–348.

Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the Workshop on Text Summarization Branches Out, 2004 2004, 74–81.

Lubis, S. S. W. (2020). MEMBANGUN BUDAYA LITERASI MEMBACA DENGAN PEMANFAATAN MEDIA JURNAL BACA HARIAN. Pionir: Jurnal Pendidikan, 9(1).

Mulasih, & Dwi Hudhana, W. (2020). URGENSI BUDAYA LITERASI DAN UPAYA MENUMBUHKAN MINAT BACA. Lingua Rima: Jurnal Pendidikan Bahasa Dan Sastra Indonesia, 9(2).

Ng, J.-P., & Abrecht, V. (2015). Better Summarization Evaluation with Word Embeddings for ROUGE. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1925–1930. https://github.com/ng-j-p/rouge-we

Rahmawati. (2020). DIKLUS: Jurnal Pendidikan Luar Sekolah. DIKLUS: Jurnal Pendidikan Luar Sekolah, 4(2), 158.

Siino, M., Tinnirello, I., & La Cascia, M. (2024). Is text preprocessing still worth the time? A comparative survey on the influence of popular preprocessing methods on Transformers and traditional classifiers. Information Systems, 121. https://doi.org/10.1016/j.is.2023.102342

Sugiarto, & Farid, A. (2023). Literasi Digital Sebagai Jalan Penguatan Pendidikan Karakter Di Era Society 5.0. Cetta: Jurnal Ilmu Pendidikan, 6(3), 580–597.

Sulfa Saguni, D., & Rahmayanti Yusuf, N. (2023). KINERJA DINAS PERPUSTAKAAN KOTA MAKASSAR DALAM MENINGKATKAN MINAT BACA MASYARAKAT THE PERFORMANCE OF THE LIBRARY OFFICE OF MAKASSAR CITY IN INCREASING PUBLIC READING INTEREST. Jurnal Administrasi Negara, 29(1).

Tahmidaten, L., & Krismanto, W. (2020). Permasalahan Budaya Membaca di Indonesia (Studi Pustaka Tentang Problematika & Solusinya). Scholaria: Jurnal Pendidikan Dan Kebudayaan, 10(1), 22–33.

Tay, W., Joshi, A., Zhang, X., Karimi, S., & Wan, S. (2019). Red-faced ROUGE: Examining the Suitability of ROUGE for Opinion Summary Evaluation. In M. Mistica, P. Massimo, & A. MackinlaY (Eds.), Proceedings of the 17th Annual Workshop of the Australasian Language Technology Association (pp. 52–60). Australasian Language Technology Association.

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., Von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., … Rush, A. M. (2020). Transformers: State-of-the-Art Natural Language Processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38–45. https://github.com/huggingface/

Yuliska, Y., & Syaliman, K. U. (2020). Literatur Review Terhadap Metode, Aplikasi dan Dataset Peringkasan Dokumen Teks Otomatis untuk Teks Berbahasa Indonesia. IT Journal Research and Development, 5(1), 19–31. https://doi.org/10.25299/itjrd.2020.vol5(1).4688

Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2020, December 18). PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. International Conference on Machine Learning. http://arxiv.org/abs/1912.08777

Zhang, T., Ladhak, F., Durmus, E., Liang, P., Mckeown, K., & Hashimoto, T. B. (2024). Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics, 12, 39–57. https://doi.org/10.1162/tacl

Zouhar, V., Meister, C., Gastaldi, J. L., Du, L., Sachan, M., & Cotterell, R. (2023). Tokenization and the Noiseless Channel. http://arxiv.org/abs/2306.16842

Downloads


Crossmark Updates

How to Cite

Kartamanah, F. F., Atmadja, A. R., & Budiman, I. (2025). Analyzing PEGASUS Model Performance with ROUGE on Indonesian News Summarization. Sinkron : Jurnal Dan Penelitian Teknik Informatika, 9(1), 31-42. https://doi.org/10.33395/sinkron.v9i1.14303