The results on Vandermonde-like matrices were introduced as a generalization of polynomial Vandermonde matrices, and the displacement structure of these matrices was used to derive an inversion formula. In this paper we first present a fast Gaussian elimination algorithm for the polynomial Vandermonde-like matrices. Later we use the said algorithm to derive fast inversion algorithms for quasiseparable, semiseparable and well-free Vandermonde-like matrices having O(n2) complexity. To do so we identify structures of displacement operators in terms of generators and the recurrence relations(2-term and 3-term) between the columns of the basis transformation matrices for quasiseparable, semiseparable and well-free polynomials. Finally we present an O(n2) algorithm to compute the inversion of quasiseparable Vandermonde-like matrices.