Multilingual-E5 Size Comparison on Indonesian Public Nutrition Regulation: A Single-Document Case Study
DOI:
10.33395/sinkron.v10i4.16807Keywords:
Dense Retrieval, Embedding Model, Indonesian Legal Corpus, Multilingual-E5, Retrieval-Augmented GenerationAbstract
Rigid structure and complex legalese frequently make it difficult to access Indonesian legal documents, such as Ministry of Health rules, and traditional Large Language Models are still prone to hallucinations. This problem is mitigated by RAG, but its accuracy is highly dependent on the semantic quality of the dense retriever, an aspect that is rarely investigated for Indonesian regulatory text, where previous studies either tested different model families on non-Indonesian corpora or used generic embeddings without efficiency trade-off analysis. In order to ascertain whether model size influences retrieval accuracy and answer quality, this study empirically benchmarks two variants of the same embedding family, multilingual-e5-large and multilingual-e5-base, within a RAG pipeline for question answering on Indonesian public nutrition law across three retrieval depths (k = 3, 5, 7). In addition to Context Relevance, Faithfulness, and Answer Relevance assessed by an LLM, 22 test questions were assessed across 132 combinations utilizing Precision, Recall, and F1-Score against ground truth. Wilcoxon signed-rank, Friedman, post-hoc, and non-inferiority tests were then used for analysis. Multilingual-e5-large and multilingual-e5-base did not differ significantly on any of the six metrics (p>0.05), although k had a significant impact on Precision, Recall, and F1-Score (p<0.05) but not the LLM-judge measures. The smaller multilingual-e5-base was non-inferior to multilingual-e5-large within a margin of 0.05 on all metrics except Recall and Faithfulness, and k can follow precision or recall priorities. Because the study covers a single regulation and computational cost was not measured, the findings should not be generalized to Indonesian legal corpora.
Downloads
References
Baur, D., Ansorg, J., Heyde, C.-E., & Voelker, A. (2025). Development and Evaluation of a Retrieval-Augmented Generation Chatbot for Orthopedic and Trauma Surgery Patient Education: Mixed-Methods Study. JMIR AI, 4, e75262. https://doi.org/10.2196/75262
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D. M., & Aletras, N. (2022). LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. 1, 4310–4330.
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated Evaluation of Retrieval Augmented Generation.
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024). A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 6491–6501. https://doi.org/10.1145/3637528.3671470
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv:2312.10997). arXiv. https://doi.org/10.48550/arXiv.2312.10997
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., … Ma, Z. (2024). The Llama 3 Herd of Models (arXiv:2407.21783). arXiv. https://doi.org/10.48550/arXiv.2407.21783
Hallee, L., Kapur, R., Patel, A., Gleghorn, J. P., & Khomtchouk, B. B. (2025). Contrastive learning and mixture of experts enables precise vector embeddings in biological databases. Scientific Reports, 15(1), 14953. https://doi.org/10.1038/s41598-025-98185-8
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. In B. Webber, T. Cohn, Y. He, & Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 6769–6781). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.550
Khalila, Z., Nasution, A. H., Monika, W., Onan, A., Murakami, Y., Radi, Y. B. I., & Osmani, N. M. (2025). Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models. International Journal of Advanced Computer Science and Applications, 16(2). https://doi.org/10.14569/IJACSA.2025.01602134
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023). MTEB: Massive Text Embedding Benchmark. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2014–2037. https://doi.org/10.18653/v1/2023.eacl-main.148
Murtiyoso, M., Tahyudin, I., & Berlilana, B. (2025). A Systematic Review of Retrieval-Augmented Generation for Enhancing Domain-Specific Knowledge in Large Language Models. Sinkron, 9(2), 969–977. https://doi.org/10.33395/sinkron.v9i2.14824
Rao, A., Alipour, H., & Pendar, N. (2025). Rethinking Hybrid Retrieval: When Small Embeddings and LLM Re-ranking Beat Bigger Models (arXiv:2506.00049). arXiv. https://doi.org/10.48550/arXiv.2506.00049
Rina, W., Km, S., Kl, M., Robert, D., Kes, M., Suryaningsih, C., Kep, S., Kep, M., Kep, S. T., Kes, M., Sya’diah, Y., Wenno, S. Z., Kes, M., Tahulending, A. A., Pd, S., & Kes, M. (2025). ANALISIS KEBIJAKAN KESEHATAN. PT Media Pustaka Indo.
Sembiring, A., & Santoso, H. (2026). Comparative Academic Performance Prediction in Primary Schools Using Linear Regression and Random Forest. Sinkron, 10(2), 1104–1113. https://doi.org/10.33395/sinkron.v10i2.15953
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., & Wei, F. (2024). Multilingual E5 Text Embeddings: A Technical Report (arXiv:2402.05672). arXiv. https://doi.org/10.48550/arXiv.2402.05672
Yafie, H. A., Zailani, A. U., Muttaqin, W., Daffansyah, M. S. A., & Ramzi, M. R. (2026). Academic Chatbot for Campus Information Services Using Retrieval-Augmented Generation. Sinkron, 10(3), 1330–1339. https://doi.org/10.33395/sinkron.v10i3.16088
Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z. (2025). Evaluation of Retrieval-Augmented Generation: A Survey (pp. 102–120). https://doi.org/10.1007/978-981-96-1024-2_8
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2026). Retrieval-Augmented Generation for AI-Generated Content: A Survey. Data Science and Engineering, 11(1), 1–29. https://doi.org/10.1007/s41019-025-00335-5
Downloads
|
How to Cite
Issue
Section
License
Copyright (c) 2026 Annisa Desianty, Yulianti, Hertanto Suryoprayogo
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Muhammad Khoiruddin harahap
Google Scholar 
Moraref
PKP Index
Garuda Kemdikbud
Indonesia OneSearch
OCLC Worldcat
Dimensions
Index Copernicus
Scilit
