Clinical Validation of an Offline Android Certainty Factor System for Pharyngitis Screening

Abstract

Certainty Factor (CF) expert systems for disease screening are widely developed in Indonesia, yet they are rarely validated prospectively against specialist diagnosis in patients who reflect the full spectrum of real-world presentations. A previously published offline Android expert system for acute and chronic pharyngitis reported 93.33% agreement with a single general practitioner on 30 retrospective cases that included no patients with other causes of sore throat. This study prospectively validates that system against the diagnosis of otolaryngologists as the reference standard, following the STARD 2015 reporting guideline. Consecutive patients presenting with sore throat entered their symptoms into the application before being examined by an otolaryngologist blinded to the application output. A confidence threshold was introduced so that outputs were classified as acute pharyngitis, chronic pharyngitis, or other, and performance was evaluated using a 3×3 confusion matrix, sensitivity, specificity, predictive values, F1-score, Cohen’s kappa, and ROC analysis. The robustness of expert-elicited weights was examined through Monte Carlo perturbation, and usability was measured with the System Usability Scale (SUS). Among 110 participants, overall accuracy was 71.8% (95% CI 62.8–79.4) and kappa was 0.55. Sensitivity was 89.1% for acute and 88.9% for chronic pharyngitis, with AUCs of 0.81 and 0.93. Most misclassifications involved patients with other diagnoses who were labelled as acute or chronic pharyngitis (22 of 31 errors). The mean SUS score was 68.2. The findings show that the system detects acute and chronic pharyngitis with high sensitivity but rarely recognises patients who have neither condition (sensitivity for “other” 21.4%), and identify examination-dependent signs as the principal constraint on unassisted self-screening.

GS Cited Analysis