Concrete Compressive Strength Prediction by Ensemble Machine Learning Approach

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Introduction
Concrete is one of the most used and important materials in civil engineering construction industries.Concrete is made with a combination of different ingredients namely cement, sand, and coarse aggregate.Additionally, water is used to create a bond between these ingredients.Moreover, the properties and workability of concrete are enhanced by using admixture.Thus, the properties and proportion of these parameters significantly influence concrete strength.The compressive strength of concrete fundamentally shows its inherent properties.Furthermore, the compressive strength of concrete stands as a key factor determining the structural integrity and durability of various infrastructural projects [1][2][3][4][5][6][7][8][9][10][11].Thus, the compressive strength of concrete is a critical parameter that influences the design, construction, and maintenance of various infrastructural projects, ranging from buildings to bridges.As a result, the prediction of concrete compressive strength is essential in the civil engineering construction field.The concrete compressive strength is often determined based on empirical formulas or laboratory testing [4,5].These laboratory methods are timeconsuming, resource-intensive, and prone to inaccuracies.To mitigate these limitations accurate prediction methods are crucial.Also, accurate prediction of concrete compressive strength not only ensures the safety and reliability of structures but also optimizes material usage and construction costs.Thus, machine learning can be an appropriate solution to address these limitations.In the last decades, many researchers have worked on the application of robust machine learning techniques in different civil engineering sectors as regression and classification tasks [4,12].Additionally, researchers have focused on exploring various techniques to enhance the prediction accuracy of concrete compressive strength by using machine learning (ML) techniques as well [5,9,[13][14][15][16][17].Among these techniques, ensemble learning methods have garnered significant attention due to their ability to combine multiple models to achieve superior predictive performance.However, most of the researchers applied single machinelearning techniques to predict concrete compressive strength.Hence, the main aim of this research study is to establish an ensemble learning technique to predict concrete compressive strength.

Ensemble Machine Learning Technique
The ensemble machine learning technique is a powerful technique that can improve the predictive performance of models by leveraging the strengths of multiple models.This technique can apply to both regression and classification problems.Fundamentally this technique consists of three types such as Bagging, Boosting, and Stacking ensemble techniques.These ensemble techniques can reduce bias and variation, increase model variety, and improve the interpretability of the final forecast of regression models [18].The short form of bootstrap aggregation is indicated by bagging.In this technique, the multiple models are trained independently, and prediction outcomes are combined by averaging for regression and majority vote for classification.Hence, this technique helps to reduce overfitting, increase stability, and enhance the generalization capability of the model [18][19][20].In addition, Boosting is another ensemble technique that combines the prediction performance of weak learners in sequential order.This technique initially trains base learners in the entire dataset.After the first model, initially, equal weights are assigned to each instance and adjusted after each iteration.Adjustments are sequentially emphasized with misclassified instances in previous models.Finally, after training all weak learners, their predictions are combined, often through a weighted sum for regression or a weighted voting scheme for classification [18,20].Furthermore, the stacking ensemble technique fundamentally consists of two main layers namely base models and meta-models [18,20,21].In this study, different machine learning models such as support vector machine (SVM), random forest (RF), decision tree (DT), k-nearest neighbor (KNN), bagging, boosting, logistic regression, etc. are used as base models, where predictions are made on the same training dataset.The predictions of base models are used as input for the meta-model.After that, the final ensemble prediction is established by combining the best prediction outcomes from different base models.Thus, this stacking technique shows better prediction performance capability, and high flexibility in model diversity acceptance in comparison with a single model.

Prediction Model Development
This research involved gathering 776 sets of data on concrete compressive strength from various past studies [9-11, 13-16, 22-31].The concrete compressive strength (CS) was predicted by taking different input variables namely cement (C), sand (S), coarse aggregate (CA), water (W), and curing time (T).Additionally, fly ash (FA) and superplasticizer were also selected as input variables.These input parameters significantly influence the prediction performance of the target variable.Table 1 outlines the statistical breakdown of these datasets.After the selection of these input and target parameters, firstly all dataset was precisely preprocessed with data cleaning, correlation analysis, data distribution, and rescaling.Afterward, the preprocessed dataset was split into training and testing sets of 80 % and 20 %, respectively.After that, the selected model was tarin with corresponding optimal hyperparameters.The optimal hypermeters of all models were established by using the grid search method with a 5-fold crossvalidation (CV) method.After that, the train performance of regression models was validated by using a testing set.The outcomes were evaluated by using different statistical indicators presented in section 3.6.If the outcomes are not in satisfactory range, then tune selected hyperparameters.In contrast, if prediction outcomes are in satisfactory range, then compare these results and establish a final prediction regression model.Figure 1 presents the detailed research methodology that is used in this research work.

Data Correlation Analysis
In this study, the correlation between selected input features and target variables was evaluated by using the Pearson correlation coefficients method.In this analysis, the range of correlation lies between -1 to 1.The high negative value is revealed with -1 value and vice versa.Figure 2 shows that compressive strength is positively correlated with cement (C), superplasticizer (SP), and curing time (T) with values of 0.29, 0.14, and 0.54, respectively.That means compressive strength is increased with an increase in the quantity of these ingredients.On the other hand, this compressive strength is negatively correlated with water content (W), sand (S), coarse aggregate (CA), and fly ash (FA) with values of 0.33, 0.14, 0.09, and 0.01, respectively.Thus, the compressive strength of concrete will decrease with use of these ingredients.Eventually, this correlation analysis underscores the importance of these parameters in accurately forecasting concrete compressive strength.Thus, all selected parameters were considered for further analysis.

Data Normalization
The selected data sets in this research study contain different scales in input parameters presented in Table 1.Thus, rescaling is essential to keep the uniform scale in all parameters.In addition, the presence of outliers is considered for further regression analysis.
In this perspective, min-max normalization was applied for rescaling and to maintain a uniform scale in selected parameters.The following equation ( 1) was used for rescaling.
Where X represents the actual value of selected parameters, Xn represents rescaled values.Additionally, Xmax and Xmin correspond to the maximum and minimum values of each input feature, respectively.

Hyper-Parameters Tuning
In this research, firstly the range of hyperparameters was selected by using the grid search method.After that, a 5-fold cross-validation technique was adopted to check performance and tune the optimal hyperparameters.Table 2 demonstrates the summary of optimal hyperparameters used in this research.For the stacking ensemble classifier, its hyper-parameters were established based on the optimization results of the base models it comprises.These optimal hyper-parameters were then employed to configure each regression model before training the model.Additionally, apart from the optimized hyper-parameters, the remaining initialization hyperparameters for each base and meta-regression model were set to their default values as defined in the Scikit-learn libraries.

Base And Meta-Model Selection
In the stacking ensemble technique, base models were selected namely decision tree (DT), support vector machine (SVM), random forest (RF), k-nearest neighbor (KNN), XGBoost, and Bagging.The optimal hypermeters of all these base regression models were established by using grid search method with a 5-fold cross-validation (CV) method.Subsequently, prediction outcomes of these base models were utilized for further analysis via optimal meta-model.Equally, these base models were utilized as meta-models, and outcomes of these metamodels were compared, and the final optimal metamodel was established.After that, the prediction outcomes of base models were used as features to train the optimal meta-model.Figure 4 illustrates the performance of each selected meta-model.Output results indicated that the random forest (RF) shows the best results as compared to others.The model shows the value of R 2 and mean squared error as 0.88, and 22.06, respectively.These results are good as compared to other meta-models.Thus, this model was used as a meta-model in this study for further analysis of the stacking ensemble regression model.

Performance Evaluation Statistical Indicators
In machine learning regression analysis, the output performance of regression analysis has been evaluated by using different statistical indicators.These indicators help researchers, data scientists, and analysts understand how well their models are performing on unseen data.Thus, these statistical indicators are crucial for assessing the effectiveness and accuracy of predictive models.In this research, the following equations ( 2) to ( 9) were applied to evaluate the effectiveness of model hyperparameters, the performance of the regression model, and in comparing different regression models.Eventually, these metrics demonstrate the effectiveness of the stacking ensemble technique over other base models.

Results and Discussion
In this study, Bagging, XGBoost, and Stacking ensemble learning models were analyzed with optimal hyperparameters.These hyperparameters were established by the Grid search method with fivefold cross-validation techniques.In the case of the Stacking ensemble regression model, the final meta-model was established with the performance of different optimized base models.Thus, in this study, the RF model was used as a meta-model.The outcomes of each ensemble model are described in the following sections.

Bagging Ensemble Model
The Bagging ensemble model was analyzed with optimal hyperparameters presented in Table 2.The output results are presented in Figure 5 and Figure 6.In  Additionally, this model reveals a good correlation between the actual values and outcomes of forecasted results, which is presented in Figure 6.This model shows an R-squared value of 0.92, which indicates that this model has good forecasting capabilities.Moreover, other statistical indices are presented in Table 3.

Boosting Ensemble Model
Boosting is a robust ensemble technique widely used for regression and classification problems.In this study, the XGBoost ensemble model was analyzed with optimal hyperparameters, which are presented in Table 2.In addition, the output results of this model are presented in Figure 7 and   3.

Stacking Ensemble Model
In this study, the Stacking ensemble model was analyzed with different base models and RF meta-models with their optimal hyperparameters.These hyperparameters are presented in Table 2.In addition, the output results of this stacking ensemble regression model are presented in Figure 9 and Figure 10.
In Figure 9, predicted results are indicated by the green line, and actual values of concrete compressive strengths are represented by the black line.These results reveal that this Stacking ensemble model exhibits good prediction capabilities for forecasting the concrete compressive strength.3.

Results Comparison
The prediction output results were comprehensively demonstrated in comparison with the actual results in the above section.These comparisons depicted that ensemble machine learning techniques have good fitting to predict the compressive strength of concrete.The detailed statistical parameters of each ensemble regression model are presented in Table 3.
In this  Therefore, this research presented that ensemble techniques significantly improve the prediction performance as compared to the individual non-ensemble regression models.In comparison with all ensemble techniques, the XGBoost shows a good prediction capability to predict concrete compressive strength.Thus, this technique might be applicable to forecast concrete strength in real construction field and further research work as well.

Figure 2 :
Figure 2: Correlation between input parameters and target variable

Figure 3
Figure 3 presents the visualization and distribution of the selected dataset in a combined form of a boxwhisker plot and violin plot.The Figure shows the statistical ranges of the dataset with its central tendency and occurrence of outliers.It seems some features namely water (W), sand (S), coarse aggregate (CA), curing time (T), and compressive strength (CS) have some outliers.In the machine learning regression model, the removal of outliers increases the performance of the model, in contrast, this removal may increase uncertainty in the real construction field.

Figure 3 :
Figure 3: Data distribution in box-whisker and violin plot Thus, the presence of outliers is selected in this research analysis.In addition, the Figure reveals the probability of occurrence by the density curve.It seems the compressive strength of the selected

Figure 5 ,
predicted results are indicated by blue lines, and actual values of concrete compressive strengths are represented by black lines.Predicted results of the Bagging model exhibit good fitting with actual values.

Figure 5 :
Figure 5: Bagging regression model forecasted outcomes comparison with actual compressive strength

Figure 6 :
Figure 6: Correlation between Bagging regression model forecasted outcomes with actual strength

Figure 8 .
In Figure 7, predicted results are indicated by a pink line, and actual values of concrete compressive strengths are represented by a black line.These results reveal that this ensemble model exhibits good prediction capabilities for forecasting the concrete compressive strength.

Figure 7 :
Figure 7: XGBoost regression model forecasted outcomes comparison with actual concrete compressive strength Additionally, this model reveals a good correlation between the actual values and outcomes of forecasted results, which is presented in Figure 8.This model shows an R-square value of 0.93, which indicates that this model has good forecasting capabilities.Moreover, other statistical indices are presented in Table3.

Figure 8 :
Figure 8: Correlation between XGBoost forecasted outcomes with actual compressive strength

Figure 9 :
Figure 9: Stacking regression model forecasted outcomes comparison with actual compressive strength Additionally, this model reveals a good correlation between the actual values and outcomes of forecasted results, which is presented in Figure 10.This model shows R-square value of 0.92, which indicates that this model has good forecasting capabilities.Moreover, other statistical indices are presented in Table3.

Figure 10 :
Figure 10: Correlation between Stacking regression model forecasted outcomes with actual compressive strength

Table 1 :
Statistical indices of selected features

Table 3 :
Regression model forecasted outcomes compari-In this study, different ensemble techniques namely Bagging, XGBoost, and Stacking were selected to predict the compressive strength of concrete.For this purpose, 776 number of datasets were collected from previously published research papers.The output results of these ensemble techniques show good predictive capability.In addition, prediction results establish a good correlation with the testing dataset.The comparison results of R 2 and VAF depict that the XGBoost has the highest values of 0.93, and 93.25 %, followed by Stacking with values of 0.92, and 91.99%, and Bagging with values of 0.92, and 91.86 %, respectively.In case of errors, the XGBoost has a minimum error in comparison with other Sacking and Bagging models.Eventually, it concludes that the XGBoost ensemble exhibits good prediction results as compared other two ensemble models.