We define a notion of finite type invariants for links with a fixed linking matrix. We show that Milnor's triple link homotopy invariant is a finite type invariant, of type 1, in this sense. We also generalize the approach to Milnor's higher order homotopy invariants and show that they are also, in a sense, of finite type. Finally, we compare our approach to another approach for defining finite type invariants within linking classes.