The optical conductivities of graphene layers are strongly dependent on their stacking orders. Our first-principle calculations show that, while the optical conductivities of single-layer graphene (SLG) and bilayer graphene (BLG) with Bernal stacking are almost frequency-independent in the visible region, the optical conductivity of twisted bilayer graphene (TBG) is frequency-dependent, giving rise to additional absorption features due to the band folding effect. Experimentally, we obtain from contrast spectra the optical conductivity profiles of BLG with different stacking geometries. Some TBG samples show additional features in their conductivity spectra, in full agreement with our calculation results, while a few samples give universal conductivity values similar to that of SLG. We propose that those variations of optical conductivity spectra of TBG samples originate from the difference between the commensurate and incommensurate stackings. Our results reveal that the optical conductivity measurements of graphene layers indeed provide an efficient way to select graphene films with desirable electronic and optical properties, which would greatly help the future application of those large-scale misoriented graphene films in photonic devices.