Pragmatic testing depends on a variety of factors that can impact its dependability. This study intended to examine these factors using a two-phase approach. The first phase examined the impact of test methods, items, raters, and test-takers’ characteristics on variance in scores of pragmatic tests by applying generalizability theory, and the second phase explored potential rater bias by applying Many-Facet Rasch model. Two test types, including a Written Discourse Completion Test (WDCT) and a Discourse Self-Assessment Test (DSAT), were administered to 110 (98 female, 12 male) English language students aged 17-24 at Vali-e-Asr University of Rafsanjan. Four raters scored the WDCT by using a standardized rubric developed by Lui (2004). The DSAT was self-assessed by test takers based on the same rubric. The findings revealed that there is no significant difference between the WDCT and DSAT test types. However, items and the interaction between items and test takers emerged as significant contributors to the variance of the scores.This highlights the importance of item calibration and rater training to mitigate bias in pragmatic testing. Finally, the implications were discussed.
Shahi, R. and Ravand, H. (2026). Dependability and Bias Analysis in WDCT and DSAT: An Application of Many Facet Rasch Model and Generalizability Theory. Teaching English as a Second Language Quarterly, 45(1), 1-28. doi: 10.22099/tesl.2025.52065.3370
MLA
Shahi, R. , and Ravand, H. . "Dependability and Bias Analysis in WDCT and DSAT: An Application of Many Facet Rasch Model and Generalizability Theory", Teaching English as a Second Language Quarterly, 45, 1, 2026, 1-28. doi: 10.22099/tesl.2025.52065.3370
HARVARD
Shahi, R., Ravand, H. (2026). 'Dependability and Bias Analysis in WDCT and DSAT: An Application of Many Facet Rasch Model and Generalizability Theory', Teaching English as a Second Language Quarterly, 45(1), pp. 1-28. doi: 10.22099/tesl.2025.52065.3370
CHICAGO
R. Shahi and H. Ravand, "Dependability and Bias Analysis in WDCT and DSAT: An Application of Many Facet Rasch Model and Generalizability Theory," Teaching English as a Second Language Quarterly, 45 1 (2026): 1-28, doi: 10.22099/tesl.2025.52065.3370
VANCOUVER
Shahi, R., Ravand, H. Dependability and Bias Analysis in WDCT and DSAT: An Application of Many Facet Rasch Model and Generalizability Theory. Teaching English as a Second Language Quarterly, 2026; 45(1): 1-28. doi: 10.22099/tesl.2025.52065.3370
)