Skip to main content
Article
Evaluating Forecasting Methods by Considering Different Accuracy Measures
Procedia Computer Science
  • Nijat Mehdiyev
  • David Lee Enke, Missouri University of Science and Technology
  • Peter Fettke
  • Peter Loos
Abstract

Choosing the appropriate forecasting technique to employ is a challenging issue and requires a comprehensive analysis of empirical results. Recent research findings reveal that the performance evaluation of forecasting models depends on the accuracy measures adopted. Some methods indicate superior performance when error based metrics are used, while others perform better when precision values are adopted as accuracy measures. As scholars tend to use a smaller subset of accuracy metrics to assess the performance of forecasting models, there is a need for a concept of multiple accuracy dimensions to assure the robustness of evaluation. Therefore, the main purpose of this paper is to propose a decision making model that allows researchers to identify the superiority of a forecasting technique over another by considering several accuracy metrics concurrently. A multi-criteria decision analysis approach, namely the preference ranking organization method for enrichment evaluation (PROMETHEE), was adopted to solve this problem. Bayesian Networks, Artificial Neural Networks, SVMs, Logistic Regression, and several Rule and Tree-based forecasting approaches were included in the analysis. After introducing a detailed description of accuracy measures, the performance of the prediction models are evaluated using a chosen dataset from the UCI Machine Learning Repository.

Meeting Name
Complex Adaptive Systems (2016: Nov. 2-4, Los Angeles, CA)
Department(s)
Engineering Management and Systems Engineering
Research Center/Lab(s)
Intelligent Systems Center
Comments
This research was funded in part by the German Federal Ministry of Education and Research under grant number 01IS12050 (project PRODIGY) and 01IS14004A (project iPRODICT).
Keywords and Phrases
  • Adaptive systems,
  • Artificial intelligence,
  • Bayesian networks,
  • Classification (of information),
  • Complex networks,
  • Decision making,
  • Learning systems,
  • Neural networks,
  • Accuracy measures,
  • Comprehensive analysis,
  • Confusion matrices,
  • Decision making models,
  • MCDA,
  • Multi-criteria decision analysis,
  • PROMETHEE,
  • UCI machine learning repository,
  • Forecasting,
  • Confusion Matrix
Document Type
Article - Conference proceedings
Document Version
Final Version
File Type
text
Language(s)
English
Rights
© 2016 The Authors, All rights reserved.
Creative Commons Licensing
Creative Commons Attribution-Noncommercial-No Derivative Works 4.0
Publication Date
11-1-2016
Publication Date
01 Nov 2016
Citation Information
Nijat Mehdiyev, David Lee Enke, Peter Fettke and Peter Loos. "Evaluating Forecasting Methods by Considering Different Accuracy Measures" Procedia Computer Science Vol. 95 (2016) p. 264 - 271 ISSN: 1877-0509
Available at: http://works.bepress.com/david-enke/44/