The goal of this study is to report the results of a comparative usability study conducted in 2008-2009 on two different course management systems (CMS), BlackBoard and Moodle. 135 students enrolled in the fall 2008 and spring 2009 section of Introduction to Educational Technology participated in the study (72 and 63 respectively). At the beginning of each semester, participants were randomly divided into two groups to experience different CMS’s at different times. The results indicated that Moodle was favored in all modules used during the courses. This article presents the details of this initial attempt on investigating if an open source CMS (Moodle) warrants a consideration as an alternative to the institution’s current CMS (BlackBoard).