Shrinkage Parameters for Each Explanatory Variable Found Via Particle Swarm Optimization in Ridge Regression

Eren Bas; Erol Egrioglu; Vedide Rezan Uslu; Eren Bas; Erol Egrioglu; Vedide Rezan Uslu

ISSN: 2641-3086

Trends in Computer Science and Information Technology

Research Article Open Access Peer-Reviewed

Shrinkage Parameters for Each Explanatory Variable Found Via Particle Swarm Optimization in Ridge Regression

Eren Bas^1*, Erol Egrioglu¹ and Vedide Rezan Uslu²

Author and article information

¹Department of Statistics, Faculty of Arts and Science, Forecast Research Laboratory, Giresun University, Giresun, 28200, Turkey
²Department of Statistics, Faculty of Arts and Science, University of Ondokuz Mayis, Samsun, 55139, Turkey

*Corresponding author: Eren Bas, Giresun University, Faculty of Arts and Science, Department of Statistics, Gure Campus, Giresun, Turkey, Tel: +90 454 3101400; Fax: +90 454 3101477; E-mail: eren.bas@giresun.edu.tr

DOI: 10.17352/tcsit.000005

Received: 23 February, 2017 | Accepted: 11 March, 2017 | Published: 13 March, 2017

Keywords: Ridge regression; Shrinkage parameters; Particle swarm optimization

Cite this as

Bas E, Egrioglu E, Uslu VR (2017) Shrinkage Parameters for Each Explanatory Variable Found Via Particle Swarm Optimization in Ridge Regression. Trends Comput Sci Inf Technol 2(1): 012-020. DOI: 10.17352/tcsit.000005

Abstract

Ridge regression method is an improved method when the assumptions of independence of the explanatory variables cannot be achieved, which is also called multicollinearity problem, in regression analysis. One of the way to eliminate the multicollinearity problem is to ignore the unbiased property of. $β$ Ridge regression estimates the regression coefficients biased in order to decrease the variance of the regression coefficients. One of the most important problems in ridge regression is to decide what the shrinkage parameter (k) value will be. This k value was found to be a single value in almost all these studies in the literature. In this study, different from those studies, we found different k values corresponding to each diagonal elements of variance-covariance matrix of instead of a single value of k by using a new algorithm based on particle swarm optimization. To evaluate the performance of our proposed method, the proposed method is firstly applied to real-life data sets and compared with some other studies suggested in the ridge regression literature. Finally, two different simulation studies are performed and the performance of the proposed method with different conditions is evaluated by considering other studies suggested in the ridge regression literature.

Main article text

Introduction

The functional relation between a dependent variable and more than one independent variable is examined by multiple regression analysis. The purpose of the multiple regression analysis is the creation of the best model that can predict the dependent variable by using the independent variables. For this purpose, the most common method to create the best model is ordinary least square (OLS) estimates method. In this method, the sum of error squares to be minimal is calculated to predict the parameters of the model.

There are some valid assumptions for the implementation of the multiple regression analysis. These are; the absence of multicollinearity problem among independent variables, the variance of error term must be constant for all independent variables and the covariance between error term and independent variables must be equal to zero.

One of the major problems in multiple regression analysis is multicollinearity problem. If there is a full or high degree linear relationship among independent variables, this situation is called as multicollinearity. Besides, multicollinearity has some important effects on OLS estimates of the regression coefficients. In the presence of multicollinearity, the OLS of regression coefficients have large variance. And also, the regression coefficients can be estimated incorrectly and the standard errors of regression coefficients can be found as exaggerated in the presence of multicollinearity. If the regression coefficients can be estimated incorrect, it can be obtained incorrect results statistically.

Therefore, ridge regression method is used to obtain stable coefficient estimates for the estimation of the regression coefficients. That means, ridge regression has been suggested to overcome the multicollinearity problem.

In the literature, it is commonly accepted that if the variance inflation factors (VIF) values are greater than 10 there is a multicollinearity problem. This is a rule of thumb and this is not exact information. Similarly, condition number can be used to determine multicollinearity problem by using rule of thumbs. As a result of, determining of multicollinearity problem can be realized by using some criteria.

The two methods most commonly used to determine the effects of multicollinearity problem are VIF and condition number methods. The diagonal elements of $V a r \hat{(β)}$ are called as VIF and are given by the Equation 1.

$V I F_{j} = \frac{1}{(1 - R_{j}^{2})} j = 1, \dots, p (1)$

In this Equation, $R_{j}^{2}$ is the determination coefficient obtained from the multiple regression of $X_{j}$ on the remaining $(p - 1)$ regressor variables in the model.

It can be said that there is a multicollinearity problem among the relevant independent variables if these VIF values increase (VIF values ≥ 10). And also, if VIF values are increased, the degree of the multicollinearity increases with the increase of VIF values.

Condition number method is another method to determine the multicollinearity problem which is based on the eigenvalues of matrix. The formula of the condition number (CN) was given in Equation 2.

$ϕ = \frac{λ_{max}}{λ_{min}} (2)$

In this Equation, $λ$ shows the eigenvalues of $X^{'} X .$ the relationship between condition number and multicollinearity is given in Table 1.

In summary, the determining of multicollinearity problem can be done by following two rules of thumbs. The first one is that if VIF values are greater than 10 multicollinearity is high. The second one is checking condition number as given in Table 1.

In addition, another problem in ridge regression is finding optimal biasing parameter (k) value. This k value is a very small constant determined by the researcher [1]. Several methods were proposed for finding it in the literature. These methods have been proposed in the studies of [2-22].

And also, there are many methods in the literature for ridge regression [23-29]. And also, [30] proposed some new methods that take care of the skewed eigenvalues of the matrix of explanatory variables. [31] Proposed an iterative approach to minimize the mean squared error in ridge regression. [32] Proposed new ridge parameters for ridge regression. [33] Proposed an optimal estimation for the ridge regression parameter. [34,35] Proposed some new estimators for estimating the ridge parameter.

This k value was found to be a single value in almost all these studies in the literature. But in this study, we found different k values corresponding to each diagonal elements of variance-covariance matrix of instead of a single value of k by using a new algorithm based on particle swarm optimization.

The rest part of the paper can be outlined as below: The second section of the paper is about ridge regression. The methodology of the paper is given in Section 3. The implementation of our proposed method is given in Section 4. Two different simulation studies are performed under the title of simulation study and finally, discussions are presented in Section 6.

Ridge regression

Ridge regression is a remedy used in the presence of multicollinearity problem and it was firstly proposed by [1]. Ridge regression method has two important advantages according to OLS method. One of them is to solve the multicollinearity problem and the other one is to decrease the mean square error (MSE). The solution technique of ridge regression is similar with OLS. Besides, the difference between ridge regression and OLS is the k value. This k value is also called as biased parameter or shrinkage parameter and it takes values between 0 and 1. This k value is added to the diagonal elements of the correlation matrix and thus biased regression coefficients are obtained.

The OLS estimates of regression coefficients and ridge estimates of regression coefficients are shown in the Equations 3 and 4 respectively.

$\begin{array}{l} \hat{β} = {(X' X)}^{- 1} X' Y \\ {\hat{β}}_{R} = {(X' X + k I)}^{- 1} X' Y (3) \end{array}$

${\hat{β}}_{R} = {(X' X + k I)}^{- 1} X' Y (4)$

As noted above, ridge regression is a biased regression method. The proof of this situation is shown in Equation 5.

$\begin{array}{l} {\hat{β}}_{R} = {(X' X + k I)}^{- 1} X' Y \\ = {(X' X + k I)}^{- 1} (X' X) \hat{β} = Z \hat{β} (5) \end{array}$

$E ({\hat{β}}_{R}) = E (Z \hat{β}) = Z β$

It is clearly seen that ridge estimates of regression coefficients $({\hat{β}}_{R})$ are biased estimates. One of the most important points to be considered in the ridge regression is the k value. There are many methods proposed in the literature to find the optimal k value. Ridge trace is one of these methods. Ridge trace is a plot of the elements of the ridge estimator versus k usually in the interval (0, 1) [1].

The other methods in the literature used to find the optimal k value were given in the Equations 6-14, respectively.

$k = \frac{ρ {\hat{σ}}^{2}}{\hat{β}' \hat{β}} (6)$

$k = \frac{ρ {\hat{σ}}^{2}}{\sum_{i = 1}^{p} λ_{i} {\hat{β}}_{i}^{2}} (7)$

$k = \frac{ρ {\hat{σ}}^{2}}{\sum_{i = 1}^{p} {{\hat{β}}_{i}^{2} [1 + (1 + λ_{i} {({\hat{β}}_{i}^{2} / {\hat{σ}}^{2})}^{1 / 2})]}} (8)$

$k = \frac{(λ_{m a x} {\hat{σ}}^{2})}{((n - p - 1) {\hat{σ}}^{2} + λ_{m a x} {\hat{β}}^{2}_{m a x})} (9)$

$k = m a x (0, \frac{p {\hat{σ}}^{2}}{\hat{β}' \hat{β}} - \frac{1}{n {(V I F_{j})}_{m a x}}) (10)$

$k = \frac{{\hat{σ}}^{2} \sum_{i = 1}^{p} (λ_{i} {\hat{β}}_{i}^{2})}{{[s \sum_{i = 1}^{p} (λ_{i} {\hat{β}}_{i}^{2})]}^{2}} (11)$

$k = \frac{{{\hat{σ}}^{2} λ_{m a x} \sum_{i = 1}^{p} (λ_{i} {\hat{β}}_{i}^{2}) + {[\sum_{i = 1}^{p} (λ_{i} {\hat{β}}_{i}^{2})]}^{2}}}{λ_{m a x} \sum_{i = 1}^{p} (λ_{i} {\hat{β}}_{i}^{2})} (12)$

$k = m a x (\frac{{\hat{σ}}^{2}}{{\hat{β}}_{i}^{2}} + \frac{1}{λ_{i}}), i = 1, 2, \dots, p (13)$

$k = \frac{p {\hat{σ}}^{2}}{\sum_{i = 1}^{p} {{\hat{β}}_{i}^{2} / [{[({\hat{β}}_{i}^{4} λ_{i}^{2} / 4 {\hat{σ}}^{2}) + (6 {\hat{β}}_{i}^{4} λ_{i} / {\hat{σ}}^{2})]}^{1 / 2} - ({\hat{β}}_{i}^{2} λ_{i} / 2 {\hat{σ}}^{2})]}} (14)$

In this paper, for the purpose of comparing the results we just consider the methods of which a brief introduction is given as below.

[2] Suggested another method for finding k value which is given in Equation 15

$k = \frac{p {\hat{σ}}^{2}}{{\hat{β}}^{'} \hat{β}} (15)$

In this Equation ${\hat{σ}}^{2}$ and $\hat{β}$ are the OLS estimates. This method is called as fixed point ridge regression method (FPRRM).

[39] Introduced an iterative method for finding the optimal k value. In this method k is calculated in Equation 16;

$k = \frac{p {\hat{σ}}^{2} (t - 1)}{\hat{β} {(t - 1)}^{'} \hat{β} (t - 1)} (16)$

In this Equation, ${\hat{σ}}^{2} (t - 1)$ and $\hat{β} (t - 1)$ are the corresponding residual mean square and the estimate vector of regression coefficients at (t-1)th iteration, respectively. This method is called as iterative ridge regression method (IRRM).

And also, the generalized ridge regression estimator of Hoerl and Kennard [1, 40] is given in [41] by following Equations 17-20.

Let and Q be the matrices of eigenvalues and eigenvectors of. $(X' X)$ In the orthogonal version of the classical linear regression model: $Z = X Q$ , $α = Q' β$ , $\hat{α} = λ^{- 1} Z' y$ , $K = d i a g (k_{1}, k_{2}, \dots, k_{p}), k_{i} > 0$ then

$\tilde{β} = Q {(Λ + K)}^{- 1} Λ \hat{α} (17)$

$\tilde{β}$ Is the generalized ridge estimator of. $β$ Hoerl and Kennard [1, 40], have shown that the values of $k_{i}$ which minimize the MSE of regression coefficient are given by

$k_{i} = \frac{σ^{2}}{α_{i}^{2}} (18)$

And the estimation of $k_{i}$ values can be obtained by using Equation 19.

${\hat{k}}_{i} = \frac{{\hat{σ}}^{2}}{{\hat{α}}_{i}^{2}} (19)$

In [41], other estimation formulas for optimum shrinkage parameters are given below.

${\hat{k}}_{i} = \frac{λ_{i} {\hat{σ}}^{2}}{(n - k) {\hat{σ}}^{2} + λ_{i} {\hat{α}}_{i}^{2}} (20)$

Methodology

Finding the optimal k value is an important problem in ridge regression. The k values recommended in the literature were given in the previous section. And also, there are some heuristic methods such as genetic algorithms to find the optimal k value in the literature proposed by [18, 21]. And also, [22] have found the k value by using particle swarm optimization (PSO). In all these methods suggested in the literature, this k value was found as a single value. But in this study, we found different k values corresponding to each explanatory variable instead of a single value of k by using an algorithm based on particle swarm optimization. And also, this paper is the improvement form of the study of [22].

The objective function of the paper was created by considering both mean absolute percentage error (MAPE) criterion and VIF values at the same time. The aim of the objective function is to find the optimal k values by finding the VIF values less than 10 and SSE (sum of square errors) minimum, at the same time. And also, we add a parameter $(\emptyset (k))$ to the second part of the objective function. This parameter can be called as penalty parameter. If the VIF value corresponds to any explanatory variable is bigger than 10 the value of the objective function is increased. This is an effect of the penalty parameter. This is an undesirable result.

The optimization problem in the proposed method can be given in Equation 21.

Objective function:

$\min_{k_{1}, k_{2}, \dots, k_{p}} M A P E (k_{1}, k_{2}, \dots, k_{p}) + \emptyset (k_{1}, k_{2}, \dots, k_{p}) (21)$

with subject to: $0 \leq k_{1}, k_{2}, \dots, k_{p} \leq 1 (j = 1, 2, \dots, p)$

where MAPE $(k_{1}, k_{2}, \dots, k_{p})$ and $\emptyset (k_{1}, k_{2}, \dots, k_{p})$ can be defined in Equations 22 and 23 respectively.

M A P E (k_{1}, k_{2}, \dots, k_{p}) = \frac{1}{n} \sum_{i = 1}^{n} | \frac{y_{i} - {\hat{y}}_{i}}{y_{i}} | (22)

$\emptyset (k_{1}, k_{2}, \dots, k_{p}) = {\begin{matrix} 0 \forall V I F_{j} < 10, j = 1, 2, \dots, p \\ \sum_{j = 1}^{p} V I F_{j} o t h e r w i s e \end{matrix} (23)$

(p shows the number of explanatory variables.)

The optimization problem defined as in (21) was solved by using PSO in the proposed method. PSO is a popular artificial intelligence technique and it was firstly proposed by [42]. The algorithm of the proposed method is given below.

Algorithm

Step 1. The parameters such as pn $c_{1}$ , $c_{2}$ , etc., are determined. These parameters are as follows:

pn: particle number of swarm

$c_{1}$ : Cognitive coefficient

$c_{2}$ : Social coefficient interval

maxt: Maximum iteration number

w: Inertia weight

Step 2. Generate random initial positions and velocities.

The initial positions and velocities are generated by uniform distribution with (0,1) parameters. Each particle has velocities up to the number of explanatory variables and each particle has positions up to the number of explanatory variables which represents $(k_{1}, k_{2}, \dots, k_{p})$ values. $x_{m}^{t}$ Represents the position of particle m at iteration t and $v_{m}^{t}$ represents the velocity of the particle m at iteration t.

Step 3. The fitness function was defined as in (21) and the fitness values of the particles are calculated.

Step 4. Pbest and Gbest particles given in (24) and (25), respectively, are determined according to fitness values.

$P b e s t_{m}^{t} = (p m), m = 1, 2, \dots, p n (24)$

$G b e s t^{t} = (p g) (25)$

Pbest is constructed by the best results obtained in the related positions at iteration t. Gbest is the best result in the swarm at iteration t.

Step 5. New velocities and positions of the particles are calculated by using the Equations given in (26) and (27).

$v_{m}^{t + 1} = [\begin{array}{l} w \times v_{m}^{t} + c_{1} \times r a n d_{1} \times \\ (P b e s t_{m}^{t} - x_{m}^{t}) + c_{2} \times r a n d_{2} \times (G b e s t^{t} - x_{m}^{t}) \end{array}] (26)$

$x_{m}^{t + 1} = x_{m}^{t} + v_{m}^{t + 1} (27)$

Where $r a n d_{1}$ and $r a n d_{2}$ are random numbers generated from U (0,1).

Step 6. Step 3 to Step 6 is repeated until t maxt.

Step 7. The optimal values are obtained as Gbest.

Implementation

The proposed algorithm was applied to two different and well known data sets in order to investigate of the proposed method. These two data sets named “Import Data” and “Longley Data” were used to evaluate the performance of the proposed method. Import data was analyzed by [43]. The variables of “Import Data” are; imports (IMPORT-Y), domestic production (DOPROD-X1), stock formation (STOCK-X2) and domestic consumption (CONSUM-X3), all measured in billions of French francs for the years 1949 through 1959. Both Import data and Longley data were solved by using fixed point method ([2]), iterative method ([39]), [22]’s method and the algorithm proposed in this paper. In the proposed algorithm, PSO parameters were chosen as $p n = 30, w = 0.9, c_{1} = c_{2} = 2$ and $m a x t = 100$ . In the iterative ridge method the stopping criteria were chosen as $= 10^{- 6}$ . The results of each method were presented in Tables 2 and 3, respectively.

As we can see from Table 2, our proposed method has minimum SSE and MAPE values. And also there is no multicollinearity problem when “Import Data” solved by our proposed method. But, there is a multicollinearity problem when “Import Data” solved by FPRRM and IRRM methods because of the VIF values of these methods are bigger than 10. Although, other methods can give smaller SSE and MAPE values they do not still solve the multicollinearity problem. Because it is clearly seen that some VIF values of these methods are greater than 10.

As we can see from Table 3, our proposed method has minimum MAPE value when compared with other methods. But SSE value of our proposed method is not the smallest one. The SSE value of OLS is smaller than our proposed methods. But, it is clearly seen that the OLS method has multicollinearity problem when “Longley Data” solved by this method. But our proposed method has no multicollinearity problem.

As a result, finding k values for each explanatory variable gives better results than finding a single k value. And also, our proposed has no multicollinearity problem.

Simulation study

Two different simulation studies are performed in this section of the paper in order to show the performance of the proposed method in different levels of multicollinearity and standard deviation of error term and the superiority of the proposed method when compared with other methods.

The First Simulation Study: In this simulation study, the proposed method was compared with ridge regression methods given in [2,22,39] by a simulation study. The number of observations (n) was taken as 100, 500 and 1000; the standard deviation of error term was taken as 0.01 and 1 and comparisons were made for the total 6 cases. For each case, 1000 data set including multicollinearity problem was created.

The first three independent variables were generated from standard normal distribution as given in Equation 28.

$X_{i} ~ N (0, 1) i = 1, 2, 3 (28)$

The last two independent variables were generated by using Equation 29. Thus, it is provided to arise multicollinearity problem for the data set by providing a high correlation between independent variables $X_{1}$ and, $X_{4}$ $X_{1}$ and $X_{5}$ .

$X_{i} = U (10, 20) + U (5, 20) X_{1} + N (0, 7) i = 4, 5 (29)$

The observations of dependent variable were obtained using Equation 30. So, all the coefficients in the regression model are taken as 1.

$Y = \sum^{} X_{i} + N (0, σ)$

For each data generated in each case, $\sum^{} V I F^{2}$ , SSE, MAPE and CN values are calculated by using proposed method, the studies [2, 22, 39]. The formula of SSE is given in Equation 31.

$S S E = \sum_{i = 1}^{n} {(y_{i} - {\hat{y}}_{i})}^{2} (31)$

The most important indicator for the comparison of methods is that VIF and CN would be small. The methods [2] and [39] do not guarantee the solution of multicollinearity problem as seen in the numerical examples. The method [22] and proposed method guarantee that all VIF values are smaller than 10. Therefore, it is suitable to compare the proposed method with [22] method in terms of SSE and MAPE criteria.

The results of median and inter quartile range (IQR) valueswere given between Tables 4-9.

When all tables are examined, it is clearly seen that and CN values of proposed method is lower than the other methods in all cases.

However, it is seen that the proposed method produces lower MAPE values compared to others despite producing higher SSE values. This is because the objective function of the proposed method may be depending to the MAPE.

The Second Simulation Study: A second simulation study was performed in the paper according to different levels of multicollinearity problem and standard deviation of error term. The regressors were generated by using Equations 32-36 given by [44].

$w_{i j} ~ N (0, 1); i = 1, 2, \dots, n; j = 1, 2, \dots, 6 (32)$

$\begin{array}{l} x_{i j} = (^{1 - ρ^{2}) 1 / 2} w_{i j} + ρ w_{i, 6}; \\ i = 1, 2, \dots, n; j = 1, 2, 3 (33) \end{array}$

$x_{i j} = w_{i j}, i = 1, 2, \dots, n; j = 4, 5 (34)$

$e_{i} ~ N (0, σ); i = 1, 2, \dots, n (35)$

$y_{i} = \sum_{j = 1}^{5} β_{j} x_{i, j} + e_{i}; i = 1, 2, \dots, n (36)$

Where $w_{i, j}$ independent standard normal are pseudorandom numbers and $ρ^{2}$ is theoretical correlation between any two explanatory variables.

Simulation study was conducted for a total of 8 cases for sample size is 100, $(n = 100)$ , standard deviation of the standard deviation of error term () and different degrees of multiple connections ( $σ = 0.01, 0.1, 1, 5$ ) $ρ = 0.99, 0.999$ (Tables 10-17).

It is clearly seen that in the tables of the simulation Study 2, $\sum^{} V I F^{2} π$ and CN values of the proposed method do not change significantly when standard deviation of error term values are changed. $\sum^{} V I F^{2}$ And CN values of the proposed method are increased dramatically when multicollinearity is increased. And also there is no a hardly ever change to be seen in the MAPE values of the proposed method with the reasonable standard deviation of error term values $(σ = 0.01, 0.1)$ or there is a decrease to be seen in the MAPE values of the proposed method when multicollinearity is increased.

In this simulation study, different levels of standard deviation of error term are also employed. As a result of this simulation study it is clearly seen that when standard deviation of error term value is greater than 1 and >1 the model has very big deviation from linear regression model because MAPE values are obtained about 60 and this value is not suitable. And also, it is clearly seen that in the tables of the simulation study 2, the prediction performance of the proposed is affected quite negatively when standard deviation of error term is increased.

Discussion

There are some valid assumptions to create a model in multiple regression analysis. One of them is that it should not be multicollinearity problem among independent variables. Ridge regression method is often used in the literature when there is a multicollinearity problem among independent variables.

But, ridge regression has also some problems. One of the most important problems in ridge regression is to decide what the shrinkage parameter (k) value will be. There are many studies in the literature to find the optimal k value. In these studies, this k value was found to be a single value. But in this study, we found different k values corresponding to each explanatory variable instead of a single value of k by using a new algorithm based on particle swarm optimization. And also, the proposed method was supported by two simulation studies. Besides, it is an important novelty for ridge regression literature.

In the future studies, different artificial intelligence optimization techniques can be used to find these k values for each explanatory variable.

References

Copyright

© 2017 Bas E, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

View Similar Articles

Share your thoughts and experiences

Article Alerts

Subscribe to our articles alerts and stay tuned.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Quick Enquiry

Table 1: Condition number and its effects.
Condition Number	Multicollinearity
CN <100	There is no serious multicollinearity
100 <CN <1000	Strong multicollinearity
CN <100	Severe multicollinearity exist in the data

Table 2: The comparison of *VIF* values, SSE and MAPE obtained from OLS, FPRRM, IRRM, [22] and proposed method for Import Data.
	*OLS* (k=0)		*[2]* (k=0.0016)		*[39]* (k=0.0042)		*[22]* k=(0.0090)		Proposed Ridge Method	k values obtained from Proposed Ridge Method
Variable	S.C.	*VIF*	S.C.	*VIF*	S.C.	*VIF*	S.C.	*VIF*	S.C. VIF	k values obtained from Proposed Ridge Method
X1	-0.34	186.11	-0.03	72.09	0.16	27.99	0.29	9.99	-0.07 9.03	k1	0.0190
X2	0.21	1.02	0.22	1.00	0.22	1.00	0.22	0.98	0.21 1.00	k2	0
X3	1.30	186.00	0.99	72.13	0.80	28.01	0.67	9.99	1.03 9.99	k3	0
SSE	0.0810		0.0086		0.0095		0.0103		0.0084
MAPE	0.1196		0.1097		0.1139		0.1185		0.1088
S.C.: Standardized Coefficients.

Table 3:The *VIF* Values, SSE and MAPE values obtained from OLS, FPRRM, IRRM, [22] and proposed method for Longley Data.
	*OLS* *(k=0)*		*[2]* *(k=0.0003)*		*[39]* *(k=0.0006)*		*[22]* *(k=0.0172)*		*Proposed Ridge Method*		k values obtained from Proposed Ridge Method
Variable	S.C.	*VIF*	S.C.	*VIF*	S.C.	*VIF*	S.C.	*VIF*	S.C. VIF		k values obtained from Proposed Ridge Method
X1	0.05	135.53	-0.01	87.32	-0.01	87.32	0.25	9.99	-0.002	0.59	k1	0.1374
X2	-1.01	1788.51	-0.25	472.15	-0.25	472.15	0.34	1.55	-0.001	0.00	k2	0.7274
X3	-0.54	33.62	-0.43	10.95	-0.43	10.95	-0.28	2.55	-0.37	3.03	k3	0.0140
X4	-0.20	3.59	-0.18	2.88	-0.18	2.88	-0.11	1.94	-0.15	2.17	k4	0
X5	-0.10	399.15	-0.28	180.21	-0.28	180.21	0.16	7.32	-0.10	5.94	k5	0.0295
X6	2.48	758.98	1.88	309.82	1.88	309.82	0.46	4.07	1.39	9.99	k6	0
SSE	0.0045		0.0050		0.0050		0.0123		0.0065
MAPE	0.0002		0.0753		0.0753		0.1378		0.0826
S.C.: Standardized Coefficients.

Table 4:Simulation results for n=100,
Method		[39]	[2]	[22]	Proposed Method
SSE	Median	0.00001259	0.00001259	0.00001363	0.00001576
SSE	IQR	0.00000756	0.00000756	0.00000931	0.00001270
MAPE	Median	0.00134451	0.00134451	0.00127049	0.00112317
MAPE	IQR	0.00106196	0.00106196	0.00089539	0.00059409
	Median	0.01140365	0.01140365	0.01139037	0.01127907
	IQR	0.01396126	0.01396126	0.01394571	0.01372089
CN	Median	34.35277075	34.35277075	34.33591354	34.27939908
CN	IQR	24.18839643	24.18839643	24.11194855	23.79666414

Table 5: Simulation results for n=100,
Method		[39]	[2]	[22]	Proposed Method
SSE	Median	0.11927554	0.11927554	0.12470745	0.12622844
SSE	IQR	0.07784014	0.07784014	0.08136102	0.08116543
MAPE	Median	0.13146684	0.13146684	0.12731270	0.12122859
MAPE	IQR	0.10275617	0.10275617	0.09496648	0.08890188
	Median	0.01265217	0.01265217	0.01055620	0.01031630
	IQR	0.01422383	0.01422383	0.01250313	0.01081028
CN	Median	36.49713949	36.49713949	34.82762683	34.19546839
CN	IQR	23.90916228	23.90916228	22.72993530	22.03645100

Trends in Computer Science and Information Technology

Shrinkage Parameters for Each Explanatory Variable Found Via Particle Swarm Optimization in Ridge Regression