Skip to main content
Article
These Are Not the K-mers You Are Looking For: Efficient Online K-mer Counting Using a Probabilistic Data Structure
PLoS ONE (2014)
  • Qingpeng Zhang, Michigan State University
  • Jason Pell, Michigan State University
  • Rosangela Canino-Koning, Michigan State University
  • Adina Chuang Howe, Michigan State University
  • C. Titus Brown, Michigan State University
Abstract
K-mer abundance analysis is widely used for many purposes in nucleotide sequence analysis, including data preprocessing for de novo assembly, repeat detection, and sequencing coverage estimation. We present the khmer software package for fast and memory efficient online counting of k-mers in sequencing data sets. Unlike previous methods based on data structures such as hash tables, suffix arrays, and trie structures, khmer relies entirely on a simple probabilistic data structure, a Count-Min Sketch. The Count-Min Sketch permits online updating and retrieval of k-mer counts in memory which is necessary to support online k-mer analysis algorithms. On sparse data sets this data structure is considerably more memory efficient than any exact data structure. In exchange, the use of a Count-Min Sketch introduces a systematic overcount for k-mers; moreover, only the counts, and not the k-mers, are stored. Here we analyze the speed, the memory usage, and the miscount rate of khmer for generating k-mer frequency distributions and retrieving k-mer counts for individual k-mers. We also compare the performance of khmer to several other k-mer counting packages, including Tallymer, Jellyfish, BFCounter, DSK, KMC, Turtle and KAnalyze. Finally, we examine the effectiveness of profiling sequencing error, k-mer abundance trimming, and digital normalization of reads in the context of high khmer false positive rates. khmer is implemented in C++ wrapped in a Python interface, offers a tested and robust API, and is freely available under the BSD license at github.com/ged-lab/khmer.
Keywords
  • Bacterial genomics,
  • Computer architecture,
  • Computer software,
  • Jellyfish,
  • Metagenomics,
  • Shotgun sequencing,
  • Subroutines,
  • Turtles
Publication Date
July 25, 2014
Publisher Statement
© 2014 Zhang et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Citation Information
Qingpeng Zhang, Jason Pell, Rosangela Canino-Koning, Adina Chuang Howe, et al.. "These Are Not the K-mers You Are Looking For: Efficient Online K-mer Counting Using a Probabilistic Data Structure" PLoS ONE Vol. 9 Iss. 7 (2014)
Available at: http://works.bepress.com/adina/3/