Practical Treatment of the Multicollinearity: The Optimal Ridge Method and the Modified OLS
Tyzhnenko A. G., Ryeznik Y. V.

Tyzhnenko, Alexander G., and Ryeznik, Yevgen V. (2021) “Practical Treatment of the Multicollinearity: The Optimal Ridge Method and the Modified OLS.” The Problems of Economy 1:155–168.
https://doi.org/10.32983/2222-0712-2021-1-155-168

Section: Mathematical methods and models in economy

UDC 330.43(075.8)

Abstract:
The paper discusses the applicability of the two main methods for solving the linear regression (LR) problem in the presence of multicollinearity – the OLS and the ridge methods. We compare the solutions obtained by these methods with the solution calculated by the Modified OLS (MOLS) [1; 2]. Like the ridge, the MOLS provides a stable solution for any level of data collinearity. We compare three approaches by using the Monte Carlo simulations, and the data used is generated by the Artificial Data Generator (ADG) [1; 2]. The ADG produces linear and nonlinear data samples of arbitrary size, which allows the investigation of the OLS equation's regularization problem. Two possible regularization versions are the COV version considered in [1; 2] and the ST version commonly used in the literature and practice. The performed investigations reveal that the ridge method in the COV version has an approximately constant optimal regularizer (λ_opt≈0.1) for any sample size and collinearity level. The MOLS method in this version also has an approximately constant optimal regularizer, but its value is significantly smaller (λ_opt≈0.001). On the contrary, the ridge method in the ST version has the optimal regularizer, which is not a constant but depends on the sample size. In this case, its value needs to be set to λ_opt≈0.1 (n-1). With such a value of the ridge parameter, the obtained solution is strictly the same as one obtained with the COV version but with the optimal regularizer λ_opt≈0.1 [1; 2]. With such a choice of the regularizer, one can use any implementation of the ridge method in all known statistical software by setting the regularization parameter λ_opt≈0.1(n-1) without extra tuning process regardless of the sample size and the collinearity level. Also, it is shown that such an optimal ridge(0.1) solution is close to the population solution for a sample size large enough, but, at the same time, it has some limitations. It is well known that the ridge(0.1) solution is biased. However, as it has been shown in the paper, the bias is economically insignificant. The more critical drawback, which is revealed, is the smoothing of the population solution – the ridge method significantly reduces the difference between the population regression coefficients. The ridge(0.1) method can result in a solution, which is economically correct, i.e., the regression coefficients have correct signs, but this solution might be inadequate to a certain extent. The more significant the difference between the regression coefficients in the population, the more inadequate is the ridge(0.1) method. As for the MOLS, it does not possess this disadvantage. Since its regularization constant is much smaller than the corresponding ridge regularizer (0.001 versus 0.1), the MOLS method suffers little from both the bias and smoothing of its solutions. From a practical point of view, both the ridge(0.1) and the MOLS methods result in close stable solutions to the LR problem for any sample size and collinearity level. With the sample size increasing, both solutions approach the population solution. We also demonstrate that for a small sample size of less than 40, the ridge(0.1) method is preferable, as it is more stable. When the sample size is medium or large, it is preferable to use the MOLS as it is more accurate yet has approximately the same stability.

Keywords: multicollinearity, economic correctness, economic adequacy, modified Cramer's rule, modified OLS, optimal ridge regression

Fig.: 3. Tabl.: 5. Formulae: 14. Bibl.: 27.

Tyzhnenko Alexander G. – Candidate of Sciences (Physics and Mathematics), Associate Professor, Associate Professor, Department of Mathematics and Mathematcal Methods in Economics, Simon Kuznets Kharkiv National University of Economics (9a Nauky Ave., Kharkіv, 61166, Ukraine)
Email: olersandr.tyzhnenko@m.hneu.edu.ua
Ryeznik Yevgen V. – Candidate of Sciences (Physics and Mathematics), Lecturer, Department of Mathematics, Uppsala University (buildings 1, 6 and 7, 1 Lagerhyddsvagen, Uppsala, 75106, Sweden)
Email: yevgen.ryeznik@math.uu.se

List of references in article

Tyzhnenko, A. G. “A new stable solution to the linear regression problem under multicollinearity“. Economics of Development. 2018. http://www.ed.ksue.edu.ua/ER/knt/ee182_86/e182tyz.pdf
Tyzhnenko, A. G., and Ryeznik, Y. V. “Ordinary List Squares: The Adequacy of Linear Regression Solutions under Multicollinearity and without it“. The Problems of Economy, vol. 1 (39) (2019): 217-227. DOI: 10.32983/2222-0712-2019-1-217-227
Seber, G. A. F. LR Analysis. New York: Wiley-Blackwell, 1977.
Seber, G. A. F. LR Analysis. New York: Wiley, 2003.
Spanos, A. Probability Theory and Statistical Inference: econometric modeling with observational data. Cambridge: Cambridge University Press, 1999.
Gujarati, D. N. Basic econometrics. New York: McGraw-Hill, 2002.
Wooldridge, J. M. Introductory Econometrics: Modern Approach. Ohio: South-Western, 2009.
Baltagi, B. Econometrics. New York: Springer, 2011.
Greene, W. H. Econometric Analysis. New York: Pearson, 2012.
Draper, N. R., and Smith, H. Applied Regression Analysis. New York: Wiley, 1966.
Farrar, D., and Glauber, R. R. “Multicollinearity in regression Analysis: The problem revisited“. Review of Economics and Statistics, vol. 49 (1967): 92-107.
Hoerl, A. E., and Kennard, R. W. “Ridge regression: Biased estimation for nonorthogonal problems“. Technometrics, vol. 12 (1) (1970): 55-67.
Marquardt, D. V. “Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation“. Technometrics, vol. 12 (1970): 591-612.
Blanchard, O. J. “Comment“. Journal of Business and Economic Statistics, no. 5 (1987): 449-451.
Adkins, L. C., and Hill, R. C. “Collinearity“. In Companion in Theoretical Econometrics, 256-278. Oxford: Blackwell Publishers, Ltd, 2001.
Belsley, D. A., Kuh, K., and Welsh, R. E. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York: Wiley, 1980.
Belsley, D. A. “Demeaning conditioning diagnostics through centering“. The American Statistician. vol. 38 (2) (1984): 73-77.
Rao, C. R., and Toutenberg, H. Linear Models: Least Squares and Alternatives. New York: Springer, 1999.
Spanos, A., and McGuirk, A. “The Problem of Near-Multicollinearity Revisited: erratic vs. systematic volatility“. Journal of Econometrics, vol. 108 (2002): 365-393.
Kabanichin, S. I. “Definitions and Examples of Inverse and Ill-posed Problems“. J. Inv. Ill-Posed Problems, vol. 16 (2008): 317-357.
Adkins, L. C., Waters, M. S., and Hill, R. C. Collinearity Diagnostics in gretl, Economics Working Paper Series 1506. Oklahoma: Oklahoma State University, Department of Economics and Legal Studies in Business, 2015.
Fox, J. Applied regression analysis, linear models, and related methods. Thousand Oaks, CA: Sage Publications, 1997.
Maddalla, G. S. Introduction to Economics. New York: Macmillan, 1992.
Harvey, A. C. “Some Comments on Multicollinearity in Regression“. Applied Statistics, vol. 26 (2) (1977): 188-191.
Tikhonov, A. N. “On the stability of inverse problems“. Doklady Acad. Sci. USSR, vol. 39 (1943): 176-179.
Tikhonov, A. N., and Arsenin, V. Y. Solutions of Ill-Posed Problems. New York: Winston & Sons, 1977.
Dougherty, C. Introduction to Econometrics. New York: Oxford University Press, 1992.