Time evolution of an electromagnetic field created in heavy-ion collisions strongly depends on the electromagnetic response of the quark-gluon plasma, which can be described by the Ohmic and chiral conductivities. The latter is intimately related to the chiral magnetic effect. I argue that a solution to the classical Maxwell equations at finite chiral conductivity is unstable due to the soft modes k < σχ that grow exponentially with time. In the kinematical region relevant for the relativistic heavy-ion collisions, I derive analytical expressions for the magnetic field of a point charge. I show that finite chiral conductivity causes oscillations of magnetic field at early times.