In [9], we introduced a linear statistical model of joint color changes in images due to variation in lighting and certain non-geometric camera parameters. We did this by measuring the mappings of colors in one image of a scene to colors in another image of the same scene under different lighting conditions. In this paper, we extend our model in several ways and examine its applicability to several important problems in machine vision. The extensions to our model include incorporating a model of image noise and a prior on the color flows used to explain a particular image difference. In addition, we increase the flexibility of our model by allowing color flow coefficients to vary according to a low order polynomial over the image. This allows us to better fit smoothly varying lighting conditions as well as curved surfaces without endowing our model with too much capacity. The problems we explore include shadow removal and detection as well as inference of scene geometry.