The aim of this work is to index images based on color, in domain specific databases using colors computed from the object of interest only, instead of using the whole image. The main problem in this task is the segmentation of the region of interest from the background. Viewing segmentation as a figure/ground segregation problem leads to a new approach--successful elimination of the background leaves the figure or object of interest. The background elements are eliminated using general observations true for any photograph where there is a single, prominent object of interest. First, we form a hypothesis about possible background colors and eliminate them, using an iterative algorithm which allows for backtracking in the event of erroneous selection of background colors. We then use an edge image at an appropriate scale to eliminate those parts of the image that are not in focus and do not contain significant structures. The edge information is combined with the color-based background elimination to produce object (figure) regions. We test our approach on a database of bird images. We show that in 87% of 600 bird images tested, the segmentation is sufficient to determine the colors of the bird correctly for retrieval purposes. These colors are used for indexing in a retrieval system. Retrieval experiments on a database of 1200 birds using known-item-search--where the goal is to retrieve a known target image given a query showing the same bird--show substantial improvements can be achieved by using our object-of-interest finder. The results show good retrieval performance even when the target images were significantly different from the query image in terms of both variations in the pose of the bird and background content. We also show that our framework can be used to incorporate domain-specific knowledge, resulting in correct segmentation of flower regions from a database of 1000 flower images in 86% of the images.