LLM-Assisted Annotation in Comparative Sentiment and Trend Analysis of Indonesian K-12 Education on X

Abstract

Indonesia was ranked 69th out of 81 participating nations in the 2022 PISA test results, reflecting persistent challenges in K-12 education quality. Public discourse on these challenges has grown substantially on social media, yet automated sentiment analysis of Indonesian education-related text remains difficult due to informal language and implicit sentiment expression. Prior work on Indonesian education sentiment has focused on short observation windows or a single policy, without comparing transformer-based models directly against traditional classifiers or scaling annotation with large language models. This study aims to compare traditional machine learning classifiers and transformer-based models for sentiment analysis of Indonesian K-12 education discourse on X, as well as the trend of sentiments longitudinally and key terms associated with different sentiments. The dataset comprises 38,957 tweets in the Indonesian language collected between January 2022 and October 2025 that were annotated via a multi-stage, large language model-assisted few-shot classification pipeline with self-consistency sampling tiebreaker, reaching inter-annotator agreement of 82.01% (Cohen’s kappa = 0.672). This study evaluated the accuracy and macro F1 score of four traditional classifiers, including Logistic Regression, Support Vector Machine, Naïve Bayes, and Random Forest, alongside three transformer-based models, namely IndoBERTweet, IndoBERT-base-p1, and IndoBERT-large-p1. Transformer-based models performed better than traditional classifiers based on all metrics. IndoBERTweet reached the best performance with 0.884 accuracy and 0.859 macro F1, compared to Logistic Regression as the best traditional classifier achieved 0.768 and 0.714, respectively. Negative sentiment dominated public discourse, with “kurikulum merdeka” as the most frequent term across all sentiment categories.

GS Cited Analysis