Abstract
This paper describes efforts related to the design, development, and evaluation of a game-based assessment called Awkward Annie. The Awkward Annie game focuses on English language pragmatic skills and represents a novel design that asks players to violate social norms by intentionally selecting the most inappropriate things to say to virtual colleagues. Awkward Annie was evaluated through two studies. Study 1 was a small-scale lab-based study with predominantly non-native English speakers (n = 36). Study 2 was a scaled up extension of the same design but with native English speakers on Amazon Mechanical Turk (n = 328). Both studies explored potential relations between user experience, understanding the play mechanics (i.e., being inappropriate), game performance, and English pragmatic ability. The studies also indicate cultural differences in players comfort with acting inappropriately. The results of these studies benefit both cross-cultural game design and the assessment of socio-pragmatics and pragmalinguistics. Additionally, this research advances the current state of assessment practices by investigating how games could potentially provide an alternative to traditional selected response testing.
