We propose a new fast spatio-temporal technique that detects an operation scene---a video segment corresponding to a single purpose diagnosis action or a single purpose therapeutic action. The technique utilizes (1) color contrast of the cable region and the background, (2) the new area-based coordinate system to compute spatial features, and (3) the history of locations of detected cables of the instrument in a video to discard false regions. The proposed technique and software are useful for (1) automatic documentation of diagnostic or therapeutic operations at the end of the procedure, (2) a second review for causes of complications due to these operations, and (3) a building block for an effective content-based retrieval system to facilitate endoscopic research and education. On 38 fulllength colonoscopy and upper endoscopy video files with different cable colors and four different insertion directions of the instruments, the average percentage of false positive duration is small at 2.23%. The average percentage of true positive duration is high at 92.9%. The average analysis time per image is 13 milliseconds on an inexpensive off-the-shelf PC.