Shiraz University Press Teaching English as a Second Language Quarterly (Formerly Journal of Teaching Language Skills) 2008-8191 38 1 2019 05 01 Pragmatic Criteria in the Holistic and Analytic Rating of the Disagreement Speech Act of Iranian EFL Learners by Non-native English Speaking Teachers 1 36 5517 10.22099/jtls.2020.34820.2738 EN Minoo Alemi West Tehran Branch, Islamic Azad University 0000-0001-9703-831X Mohammad Motamedi Sharif University of Technology Journal Article 2019 09 12 <strong>Conveying a strong message within a language stems from not only a linguistically appropriate utterance but also a pragmatically appropriate discourse. Broadly considering various facets of pragmatics, pragmatic assessment has not been potentially brought into perspective. To address this discourse gap, this study, guided by the principles of mixed-method design, pursued three purposes: to inspect the matches and mismatches, to explore rating variations, and to assess the rater consistency between the holistic and analytic rating methods of disagreement speech acts in L2 by non-native English teachers. As a result, 12 different pragmatic situations for disagreement DCTs accompanied by EFL learners' responses to each situation were rated by 50 non-native English teachers. Initially, they were asked to rate it holistically, incorporating both ratings and providing comments. The content analysis of raters' comments indicated sixteen disagreement criteria. The descriptive statistics also revealed variations across different situations. Moreover, the teachers were asked to rate it analytically based on the assessment rubrics adopted from Ishihara and Cohen (2010). The findings of intra-class correlations implied that respondents were more consistent in analytic rating. Moreover, the results indicated that there was a convergence between the two rating methods suggesting that the raters adopted the same level of leniency and severity in rating. Overall, the results accentuated the significance of pragmalinguistic and sociopragmatic aspects of language for EFL raters. Finally, the results of the present study place a premium on the importance of pragmatic assessment training as well as cultural awareness. </strong> https://tesl.shirazu.ac.ir/article_5517_967157c366dc0d1cc8ab3db4969ee7d6.pdf