We provide a systematic measurement of the rest-frame UV continuum slope beta over a wide range in redshift (z~2-6) and rest-frame UV luminosity (0.1-2L*) to improve estimates of the SFR density at high redshift. We utilize the deep optical and infrared data (ACS/NICMOS) over the CDF-S and HDF-N GOODS fields, as well as the UDF for our primary UBVi "dropout" sample. We correct the observed distributions for selection biases and photometric scatter. We find that the UV-continuum slope of the most luminous galaxies is substantially redder at z~2-4 than it is at z~5-6. Lower luminosity galaxies are also found to be bluer than higher luminosity galaxies at z~2.5 and z~4. We do not find a large number of galaxies with beta's as red as -1 in our dropout selections at z~4, and particularly at z>~5, even though such sources could be readily selected from our data. This suggests that star-forming galaxies at z>~5 almost universally have very blue UV-continuum slopes, and that there are not likely to be a substantial number of dust-obscured galaxies at z>~5 that are missed in "dropout" searches. Using the same relation between UV-continuum slope and dust extinction as found to be appropriate at z~0 and z~2, we estimate the average dust extinction of galaxies as a function of redshift and UV luminosity in a consistent way. We find that the estimated dust extinction increases substantially with cosmic time for the most UV luminous galaxies, but remains small (<~2x) at all times for lower luminosity galaxies. Because these same lower luminosity galaxies dominate the luminosity density in the UV, the overall dust extinction correction remains modest at all redshifts. We include the contribution from ULIRGs in our SFR density estimates at z~2-6, but find that they contribute only ~20% of the total at z~2.5 and <~10% at z>~4.