Analyzing the System Usability Scale in Open-Source Information Systems: An Item-Level Psychometric Assessment of Post-Adoption Usability of KOHA ILS

Abstract

System Usability Scale (SUS) is still an industry standard tool for fast software usability testing. But its psychometric performance in the long-term post-adoption cycles, in developing countries institutional structures is less investigated. This study is based on the Usability Theory and Technology Acceptance Model (TAM), and narrows the scope to the SUS to conduct a separate psychometric diagnostics of the open source KOHA Integrated Library System (ILS). In 2025, a follow-up sample of 20 librarians with long-term experience with KOHA ILS was given the same 10 item SUS after an initial baseline assessment of the sample. To get more precise confidence intervals for the regression, 1000 resamples were used in bootstrapping, given the small sample size. The mean SUS score of the system was 61.6, which shows that the system has acceptable usability level. An item-level analysis showed that the respondents agreed on the positive rating of the system integration, but there were still some problems with regard to the system complexity and system learnability according to the rating of the negatively worded items. Based on exploratory regression analysis, there were positive correlations between user competence and interface customization and SUS scores. The results have to be considered with some caution due to the small sample size.