Skip to main content
Unpublished Paper
Unsupervised Deduplication using Cross-Field Dependencies
  • Robert Hall
  • Charles Sutton
  • Andrew McCallum, University of Massachusetts - Amherst
Recent work in deduplication has shown that collective deduplication of different attribute types can improve performance. But although these techniques cluster the attributes collectively, they do not model them collectively. For example, in citations in the research literature, canonical venue strings and title strings are dependent---because venues tend to focus on a few research areas---but this dependence is not modeled by current unsupervised techniques. We call this dependence between fields in a record a cross-field dependence. In this paper, we present an unsupervised generative model for the deduplication problem that explicitly models cross-field dependence. Our model uses a single set of latent variables to control two disparate clustering models: a Dirichlet-multinomial model over titles, and a non-exchangeable string-edit model over venues. We show that modeling cross-field dependence yields a substantial improvement in performance---a 58% reduction in error over a standard Dirichlet process mixture.
  • Information Systems,
  • Database Management,
  • Database Applications,
  • data mining,
  • information extraction,
  • deduplication,
  • Dirichlet process mixture
Publication Date
This is the pre-published version harvested from CIIR.
Citation Information
Robert Hall, Charles Sutton and Andrew McCallum. "Unsupervised Deduplication using Cross-Field Dependencies" (2008)
Available at: