A scalar intensity measure (IM) could be insufficient to represent the earthquake intensity and variety in fragility estimation. Introducing multiple IMs to conventional regression of fragility functions can be computationally demanding and require priori assumptions of functional forms. In this study, multivariate seismic classifiers with multiple IMs as inputs are developed based on artificial neural networks (ANNs) to address the above disadvantages of traditional regression approaches. Case studies of a four-story code-conforming benchmark building indicate that fragility estimates from multi-IM ANN classifiers lead to higher accuracy (5.0% to 7.7%) in system-level and element-level damage classification than the single-IM traditional fragility curves. Further studies of IM combinations show that the ANN performance can be improved by more IMs correlated with structural responses while compromised by redundant irrelevant IMs. The optimal IM set should be determined by correlation ranking and ANN predictive performance together. Moreover, the ANN configuration of the case-study building is optimized with five readily available IMs as inputs, which enable a near real-time (within 0.3 ms) prediction of future earthquake damage while maintain high predictive performance. Overall, the multivariate ANN seismic classifier can be a promising tool for simultaneous seismic fragility estimation and damage assessment.