Skip to main content
Presentation
Using Web Scraping Technology to Automatize Cost Data Collection for Calculating Construction Cost Indexes
1st Graduate and Professional Student Research Conference (2014)
  • Joseph Shrestha, Iowa State University
  • H. David Jeong, Iowa State University
Abstract
Construction Cost Index (CCI) and Building Cost Index (BCI) published by Engineering News Record (ENR) are two of the very popular cost indexes used in the industry. Many state DOTs publishes their highway construction cost indexes to keep track of the current trend of cost escalation. The ENR BCI and ENR CCI are prepared by collecting cost data from agents of the 20 cities in the country. Such manual process of collecting cost data can be costly. The proposed method of price data collection, based on a computer technology called “web scraping,” can automatically collect material prices and labor wage rates from the websites of material suppliers and labor unions. The material price data and labor rate can then be used to calculate various construction cost indexes. As a proof of concept, an application has been developed that collects price data from one of the popular chain store websites that sells the construction materials in small scale. The collected data is then stored in a MySQL database for further use. The benefit of using this technology is ability to calculate the indexes as often as required. Also, it would be easy to extend the geographical coverage of the indexes based on the availability of the suppliers publishing the cost data online for that location. The limitation of the technology is that if there are changes in the structure of the website which are beyond the changes in just a layout of the website, then the system is likely to break requiring a manual fix. 
Keywords
  • web scraping technology,
  • technology,
  • cost data collection,
  • construction cost indexes
Publication Date
January 1, 2014
Location
Iowa State University, Ames, Iowa
Comments
Abstract available through theGraduate and Professional Student Research Conferenceat Iowa State University.
Citation Information
Joseph Shrestha and H. David Jeong. "Using Web Scraping Technology to Automatize Cost Data Collection for Calculating Construction Cost Indexes" 1st Graduate and Professional Student Research Conference (2014)
Available at: http://works.bepress.com/joseph_shrestha/7/