Skip to main content
Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
  • Guanzheng Chen, Sun Yat-sen University
  • Fangyu Liu, University of Cambridge
  • Zaiqiao Meng, University of Glasgow
  • Shangsong Liang, Mohamed bin Zayed University of Artificial Intelligence
Document Type

Parameter-efficient tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs). By tuning just a fraction amount of parameters comparing to full model finetuning, PETuning methods claim to have achieved performance on par with or even better than finetuning. In this work, we take a step back and re-examine these PETuning methods by conducting the first comprehensive investigation into the training and evaluation of PETuning methods. We found the problematic validation and testing practice in current studies, when accompanied by the instability nature of PETuning methods, has led to unreliable conclusions. When being compared under a truly fair evaluation protocol, PETuning cannot yield consistently competitive performance while finetuning remains to be the best-performing method in mid- and high-resource settings. We delve deeper into the cause of the instability and observed that model size does not explain the phenomenon but training iteration positively correlates with the stability. Copyright © 2022, The Authors. All rights reserved.

Publication Date
  • Competitive performance; Evaluation of parameters; Evaluation protocol; Full model; Language model; Method claims; Performance; Protocol parameters; Tuning method

Preprint: arXiv

Citation Information
G. Chen, F. Liu, Z. Meng, and S. Liang, "Revisiting parameter-efficient tuning: Are we really there yet?" 2022, arXiv:2202.07962