Detection and characterization of rare circulating tumor cells (CTCs) in patients' blood is important for the diagnosis and monitoring of cancer. The traditional way of counting CTCs via fluorescent images requires a series of tedious experimental procedures and often impacts the viability of cells. Here we present a method for label-free detection of CTCs from patient blood samples, by taking advantage of data analysis of bright field microscopy images. The approach uses the convolutional neural network, a powerful image classification and machine learning algorithm to perform label-free classification of cells detected in microscopic images of patient blood samples containing white blood cells and CTCs. It requires minimal data pre-processing and has an easy experimental setup. Through our experiments, we show that our method can achieve high accuracy on the identification of rare CTCs without the need for advanced devices or expert users, thus providing a faster and simpler way for counting and identifying CTCs. With more data becoming available in the future, the machine learning model can be further improved and can serve as an accurate and easy-to-use tool for CTC analysis.