Detection of 3D object-motion trajectories depends on the integration of two distinct visual cues: translational displacement and looming. Electrophysiological studies have identified distinct neuronal populations, whose activity depends on the precise motion cues present in the stimulus. This distinction, however, has been less clear in humans, and it is confounded by differences in the behavioral task being performed. We analyzed whole-brain fMRI, while subjects performed a common time-to-arrival task for objects moving along three trajectories: moving directly towards the observer (collision course), with trajectories parallel to the line of sight (passage course), and with trajectories perpendicular to the line of sight (gap closure). We found that there was substantial overlap in the pattern of activation associated with each of the three tasks, with differences among conditions limited to the human motion area (hMT+), which showed greater activation extent in the gap closure condition than for either collision or passage courses. These results support a common substrate for temporal judgments of an object’s time-to-arrival, wherein the special cases of object motion directly toward, or perpendicular to, the observer represent two extremes within the broader continuum of 3D passage trajectories relative to the observer.