This paper introduces the regional assessment called SEA-PLM and explains the challenges faced by test developers in terms of construct definition, task development and scoring. Prior to the field trial, ACER test developers hypothesised that some features of writing are comparable across languages, while others are language specific. However, after removing poorly functioning items post field trial, the main study showed that all the features of writing assessed were comparable across languages. This provides the foundation for an argument that cross-language comparisons of writing literacy are possible by using generic criteria for scaling.