Skip to main content
Presentation
Economical Big Local Storage
Coalition for Networked Information (CNI) (2013)
  • Tom Klingler, Kent State University
Abstract

Kent State University Libraries has developed a local digital storage system that provides high-volume, medium-term storage of digital items. The system uses very inexpensive hardware and locally-developed (soon to be open source) software. Technically not a DAM nor a dark archive, this system provides for the distributed redundant storage of three validated (fixity checked) copies of digital files, some of which could then move to DAM or dark archive based on retention schedule and/or significance. The system supports upload via batch, zip, and drag and drop, and it supports a wide array of standard file types. Items are assigned an expiration date based on a retention period, which is based, in turn, on an assigned retention group. System-assigned and user-supplied metadata support a search mechanism. Stored content is organized by Workgroup, Project, and Item. Three servers (pods) distributed throughout campus each support 36TB of storage. The servers synchronize all files daily and nightly between themselves when all checksums are valid. The system runs on CentOS, and uses RAID6. There is no strict 7/24/365 up-time expectation; the primary concern is data preservation.

End-users, staff in various campus divisions, contribute their own data for storage, and the data in question includes data that is the responsibility of University Libraries, Special Collections, and University Archives. The system stores master files like tiffs; there is no need to store derivative files, which can be easily regenerated on the fly. At the current capacity of 36TB per each pod, the total cost is $ 0.58 per GB. The current hardware configuration can support up to 108TB per pod, at a total cost of $0.30 per GB. This system is envisioned as a middle-layer storage system that can provide massive storage at very low cost, and it can provide a central workspace where data can be stored before it is moved to other systems like a DAM system, a dark archive, an institutional repository, a public digital gallery, etc., or before it is disposed of based on a retention schedule. This session will offer more detail on the system’s hardware and software functionality.

Keywords
  • storage systems
Publication Date
April 4, 2013
Citation Information
Tom Klingler. "Economical Big Local Storage" Coalition for Networked Information (CNI) (2013)
Available at: http://works.bepress.com/tom_klingler/12/