Automated writing evaluation (AWE) entered the scene of academic writing pedagogy with a promising potential to enhance writing development through individualized formative feedback. However, despite evidence of positive impact (Stevenson, 2016), AWE technologies have been vehemently criticized because, to writing teachers, their engines running in the background are nothing but black-boxes (Herrington & Moran, 2012) that evaluate writing based on aggregated quantifiable text features (Shermis & Burstein, 2013). Writing, however, is essentially about meaning making and reflects rhetorical aspects of different social and academic genres (Perelman, 2012). This criticism requires revisiting a challenging yet foundational question for academic writing teachers: How can we, as stakeholders in AWE-assisted writing support, ensure that AWE technologies help us appropriately focus on important traits of writing as communicative practice?