Comparative Evaluation of Machine Learning Algorithms for Intrusion Detection Systems

Abstract

Performance estimates of intrusion detection models may vary depending on how the training and testing data are separated. Although random split is commonly used in IDS experiments, network traffic often follows time-dependent patterns that differ from one day to another. This study compares random split, single temporal split, and rolling temporal split to examine whether random evaluation produces overly optimistic performance estimates. The CIC-IDS2017 dataset was used because it contains network traffic collected across several days and includes benign as well as malicious activities. The evaluation involved five classical learning models: Decision Tree, Random Forest, Logistic Regression, K-Nearest Neighbors, and Linear Support Vector Machine. The dataset was prepared by combining daily traffic files, removing irrelevant and invalid features, converting labels into binary classes, and applying consistent preprocessing for all models. Performance was measured using accuracy, precision, recall, F1-score, ROC-AUC, PR-AUC, training time, and prediction time. The results show that random split produced very high scores, with several models reaching F1-scores close to 1.0. In contrast, temporal evaluation caused a clear performance decrease, with single temporal F1-scores ranging from approximately 0.60 to 0.71, while rolling temporal validation showed that model performance varied across different chronological testing periods. These findings indicate that random split may overestimate IDS model performance because similar traffic patterns can appear in both training and testing data. Therefore, time-aware evaluation provides a more realistic strategy for assessing IDS model generalization.

GS Cited Analysis