Support vector regression performance analysis and systematic parameter selection

Abstract

Support Vector Regression (SVR) based on statistical learning is a useful tool for nonlinear regression problems. The SVR method deals with data in a high dimension space by using linear quadratic programming techniques. As a consequence, the regression result has optimal properties. However, if parameters were not properly selected, overfitting and/or underfitting phenomena might occur in SVR. Two parameters sigma, the width of Gaussian kernels and epsilon, the tolerance zone in the cost function are considered in this research. We adopted the concept of the sampling theory into Gaussian Filter to deal with parameter sigma. The idea is to analyze the frequency spectrum of training data and to select a cut-off frequency by including 90% of power in spectrum. The corresponding sigma can then be obtained through the sampling theory. In our simulations, it can be found that good performances are observed when the selected frequency is near the cut-off frequency. For another parameter epsilon, it is a tradeoff between the number of Support Vectors and the RMSE. By introducing the confidence interval concept, a suitable selection of epsilon can be obtained. The idea is to use the L-1-norm (i.e., when epsilon = 0) to estimate the noise distribution of training data. When E is obtained by selecting the 90%, confidence interval, simulations demonstrated superior performance in our illustrative example. By our systematical design, proper values of sigma and epsilon can be obtained and the resultant system performances are nice in all aspects.