An integrated data-centric portfolio is presented in this work to tailor the performance of the deep learning method. The proposed portfolio includes attention mechanisms, feature-enabled image augmentation strategy, and orthogonal test-based parameter fine-tuning. It focuses on improving the quality and diversity of the data without changing the architecture of the model. The results show that by embedding attention modules into the deep learning-based model, the mAP50 is increased by 3.1% compared to the benchmark. A Class-Specific Image Augmentation (CSIA) method is proposed to work as an optimism strategy for quantifying the number of generated images for each distress. It outperforms augmenting images for all distresses equally which has been widely used in many studies. An orthogonal test method is introduced to decrease the training time for parameter fine-tuning. With the proposed data-centric portfolio, mAP50 of the YOLOv5s model is significantly improved from 0.594 to 0.818.