In this paper, the infinite horizon optimal tracking control problem is solved online and forward-in-time for leader-follower based formation control of nonholonomic mobile robots. Using the backstepping approach and the kinematic controller developed in our previous work, the dynamical controller inputs for the robots are approximated from nonlinear optimal control techniques to track the designed control velocities. The proposed adaptive dynamic programming approach uses neural networks (NNs) to solve the optimal formation control problem in discrete-time in the presence of unknown internal dynamics and a known control coefficient matrix. All NNs are tuned online using novel weight update laws, and the stability of the entire formation is demonstrated using Lyapunov methods. Numerical simulations are also provided to demonstrate the effectiveness of the proposed approach.