Social media provide users a channel to share meaningful and insightful information with their network of connected individuals. Harnessing this public information at scale is a powerful notion as social media is rife with public perceptions, signals, and data about a variety of topics. However, there is a common trade-off in collecting information from social media: the more specific the topic, the more challenging it is to extract reliable and truthful information. In this paper, we present an experience report describing our efforts in developing and applying a novel approach to identify, extract, and validate topic specific information using the Amazon Mechanical Turk (AMT) crowdsourcing platform. The approach was applied in a use-case where meaningful information about a medical condition (major depressive disorder) was successfully extracted from Twitter. Our approach, and lessons learned, may serve as a generic methodology for extracting relevant and meaningful data from social media platforms and help researchers who are interested in harnessing Twitter, AMT, and the like for reliable information discovery.