Evaluating Research Reports on the Qualities of Tests of English Language Skills in Indonesian Schools: A Systematic Review

Abstract

The teaching of English as a foreign language in Indonesia has a long history, and it is always important to ask whether the assessment of the students' language skills has been valid and reliable. A screening of many articles in several prominent databases reveal that a number of evaluation studies have been done by Indonesian scholars in the last 14 years. This paper reports a systematic review with the aim of critiquing those evaluation studies to see the soundness of their methods and their results. PRISMA framework was used to screen a large number of articles from the databases and to finally obtain 14 research papers published in various journals. The findings indicate that most of the studies were focused on the analysis of the items in multiple-choice tests, and on the content validity, reliability and construct validity of those tests. A further scrutiny revealed that many of these studies lacked methodological rigor, including the absence of expert judgment in content validation, limited application of psychometric frameworks such as Aiken's V formula, and insufficient procedures for construct validation. While the measurement of the item difficulty, item discriminatory power, and distractors' efficiency were relatively adequate, the approaches to determining the content validity, construct validity, and reliability of the tests remained overly subjective and inconsistent. These findings highlight the need for improvements in language test research practices in Indonesia, including structured training for teachers in language assessment, the adoption of psychometric-based validation methods, and systematic involvement of expert judgment in test development processes.