Reducing Overfitting Problem by Hyperparameter Optimization of Different Regularization Techniques using Linear Regression

Abstract

In machine learning, regularization refers to the techniques for calibrating learning models that prevent overfitting and lower the adjusted loss function. Regularization is useful for attaining the perfect fit between overfitting and underfitting. This research focuses on identifying the problem of overfitting on linear regression model and also aims to solve that problem by implementing different regularization technique like Lasso (L1 regularization), Ridge (L2 regularization) and Elastic-Net (L1_L2 regularization) which are applicable on regression model. Optimization of the hyperparameter is also carried out for all the regularization techniques in order to find the ideal value of the hyperparameter to minimize the MSE and to enhance the model performance and efficiency. The dataset used for this research is the Nepali Housing Price Dataset which has been downloaded from Kaggle repository. MSE is used as performance metrics for to analyze the performance of regularization techniques. From the experimental results, it is observed that the Elastic-Net regression outperformed both Lasso and Ridge regression with the minimum MSE of 3.5541 at lambda (λ) value of 0.9. Also, the MSE provided by Lasso and Ridge at same lambda (λ) value of 0.9 are 12.6478 and 4.6688. Thus, this research concluded that the Elastic-Net regression provides the minimum MSE for the dataset and also help to overcome the problem of overfitting by enhancing the overall performance and efficiency of the model.