Based on machine learning model for prediction of CO₂ adsorption of synthetic zeolite in two-step solid waste treatment

2.2.1 Support vector regression model

The Support Vector Machines (SVM) are one of the most popular supervised learning-based machine learning models. Their basic principle is to find the hyperplane that forms the largest interval in the feature space to discriminate between different types of data sets(Cortes and Vapnik, 1995). Therefore, this model was originally used to solve two-class classification problems(Suthaharan, 2016, Sunitha and Raju, 2021). For the regression problem, support vector regression (SVR) is used which is based on the support vector machine principle. The schematic diagram of the algorithm is shown in Fig. 2 (a). By continuously optimizing the hyperparameter combinations, it is acceptable to find a large number of points that are matched in the SVR and the remaining points as close as possible to within a certain error range (epsilon band in the figure), which is within a certain distance from the true value. Moreover, the introduction of various kernel functions φ(x, y) maps the low-dimensional sample space to the high-dimensional space, making it linearly separable to handle nonlinear regression problems. Therefore, different kernel functions may have a certain impact on the prediction performance of the final model(Ma et al., 2015). The current mainstream kernel functions are linear kernel, Gaussian kernel, radial basis kernel, etc. The choice of the hyperparameters on the SVR are explained in section 2.3.1.

2.2.2 Multivariate adaptive regression splines model

The multivariate adaptive regression spline (Mars) is a data analysis method proposed by American statistician Jerome Friedman in 1991(Friedman, 1991). The method uses the tensor product of the spline function as the basis function and is divided into three steps: Forward process, backward pruning process and model selection. The basic principle of the algorithm is shown in Fig. 2(b). The data is segmented by adaptively selecting nodes, and two new linear basis functions are generated by selecting each node. When the forward process is complete, N + 1 linear basis functions are used to generate an overfitting model. Under the premise of ensuring the accuracy of the model, the basis functions that contribute little to the model in the overfitting model are deleted backward. Finally, an optimal model is selected as the regression model. The choice of the hyperparameters on Mars are explained in section 2.3.1.

2.2.3 Random forest model

The random forest (RF) is an integrated model based on the Bagging algorithm and consists of basic units of decision trees. The so-called integrated model is used to obtain a learner with better generalization performance. By training several individual learners and combining certain combination strategies, the single learner model with poor performance eventually becomes a strong learner model(Cutler et al., 2012, González et al., 2015). The basic principle of the random forest model is shown in Fig. 2(c). The RF model randomly samples the original dataset to form N different sample datasets, and then builds N different decision tree models based on these datasets. Finally, the final result is obtained based on the average of these decision tree models (for the regression model) or the voting situation (for the classification model), and the final random forest model is output based on the evaluation results. The choice of the hyperparameters on RF are explained in section 2.3.1.

2.2.4 Gradient boosting machine model

Gradient Boosting Machine (GBM) is another type of integrated learning that differs from Random Forest. Although both models consist of basic decision trees, the GBM is based on the boosting algorithm(Ayyadevara, 2018, Konstantinov and Utkin, 2021). Its main principle is shown in Fig. 2(d). It consists of building a collection of shallow and weak continuous trees sequentially, where each tree learns and improves the previous tree and continuously adds new models to the collection. Thus, at each specific iteration, a new weak-base learner model is trained based on the errors of the entire ensemble learned so far. This continuously increases the accuracy of the model, which is usually difficult to beat by other algorithms. The choice of the hyperparameters on GBM are explained in section 2.3.1.

2.3.1 Model training and testing

In this study, four machine learning models were built and the hyperparameters were adjusted to optimize the model accuracy and improve the generalization ability of the model. Hyperparameter grid search in the package of CARET(Kuhn et al., 2021) randomly matches the selected hyperparameter combinations, determines different hyperparameter combinations, and trains machine learning models of different hyperparameter combination schemes that help find reasonable values for tuning parameters. At the same time, triple independent 10-fold cross-validation was used as a resampling scheme to avoid final overfitting of the model. Minimize the error between the predicted value and the experimental value with such a method. Table 2 contains all hyperparameters considered by the four machine learning models and their meaning.

2.3.2 Model performance evaluation

For the performance evaluation of four regression prediction model, the following three indexes are mainly investigated: root mean squared error (RMSE), mean absolute error (MAE), coefficient of determination (R²). The RMSE and MAE are indicators for evaluating model errors. The smaller the value is, the stronger the prediction accuracy of the model. On the contrary, the R² represents the proximity between the predicted value of the model and the measured value, and the closer the value is to 1, the higher the prediction accuracy of the model. The calculation formulas for these three indicators are as follows.

Where, Y_Ture,i, Y_Model,i, and $\overline{Y}$ denote the i_th sample’s experimental value, the i_th sample’s predicted value and the average value of all experimental values; N indicates the total number of samples.

3.1.1 Optimization of support vector machines

Fig. 3 compares the predictive performance of support vector regression models with three different kernels, including the Gaussian kernel function, the linear kernel function, and the radial basis function. The SVR of these three kernels all uses tuneLength = 5 to train the model. What needs to be explained is that the meaning of tuneLength is different for the different machine learning models. For SVR, tuneLength = 5, which means you try 5 different sigma and cost values to train the model. From Section 2.3.2, the smaller MAE and RMSE values denote the better models, while the higher R² means the better models. Therefore, the SVR model whose kernel is a radial basis kernel function has the best predictive performance among the three kernels, and the value of R² reaches 0.97. The SVR models with linear kernels and Gaussian functions do not perform well. The MAE value of the SVR model of the radial basis kernel function is 0.26, while the MAE value of the Gaussian kernel function is about 1.7 times that of the radial basis kernel function. The MAE of the linear kernel function is about 1.5 times that of the radial basis function. Compared to the radial basis function model, the RMSE of the linear kernel function and the Gaussian kernel function are 1.6 times, respectively. From the above, it can be seen that the radial basis kernel function has better accuracy than the others. At this point, the optimal value of Cost and Sigma obtained by the random search are 4 and 0.16, respectively.

Close

Fig. 3

Accuracy of SVR for different kernel functions.

Export to PPT

To further investigate and verify the influence of the value of Cost and Sigma on the accuracy of the SVR model with radial basis kernel function, the performance indicators of the SVR model under 35 sets of different Cost and Sigma value hyperparameter combinations are evaluated in Fig. 4. The meanings of the Cost and Sigma are given in Table 2. The values of Cost control the trade-off between achieving a low error on the training data and minimizing the norm of the weights. As the Cost value continues to increase, it means that the tolerance range of the model is gradually expanding and the accuracy of the model is improving, but it is also prone to overfitting. It can be seen from the figure that the accuracy of the model improves significantly as the value of Cost increases from 1 to 100. As the value of Cost continues to increase, the improvement in the model performance index becomes gradually apartment. The sigma serves as another hyperparameter defining the range of influence of a single training example. For the R² index in the SVR model, when Sigma is 0.05, the performance indicators of the model are smaller than when Sigma is 0.1. This is because the SVR model boundary at low values of Sigma is essentially based on the point closest to the boundary, ignoring the more distant points that do not exactly fit all the data, and the model boundary will be smoother. When the Sigma is greater than 0.1, the performance index of the model does not improve significantly, and the larger Sigma may lead to over-fitting of the model. The optimal combination of hyperparameters for the SVR model is shown in Table 3.

Close

Fig. 4

Optimization of Sigma and Cost to accuracy of radial basis function based SVR.

Export to PPT

3.1.2 Optimization of multivariate adaptive regression splines model

Fig. 5 shows the optimization of the accuracy considering the combination of the two hyperparameters degree and nprune. The parameter degree specifies the maximum degree of interaction, while the parameter nprune specifies the number of terms to be preserved in the final model. The degree parameter is set to 1 by default, which means that the interaction term is not considered when creating the MARS model. Although the generalizability of the model can be tested by selecting more interaction elements, an upper limit should be set on the degree of interaction. A lower degree of interaction can not only save the time needed to run the model, but also help explain the final model(Friedman et al., 2001). It can be seen from the figure that the overall generalization of the model is poor when the interaction element is not considered, and R² can only reach about 0.8. When the value of degree is 2, the R² increases by about 18 %, but as the degree increases, the more interaction elements are considered, the less the accuracy of the model improves. In addition, the performance of the model is also significantly improved by increasing the value of nprune. When the degree is 2 and nprune is 20, the R² can reach the highest value of 0.954, and the RMSE and MAE are 0.29 and 0.20, respectively. When nprune continues to increase from 20, the performance of the model will not change. The optimal combination of hyperparameters for the Mars model is shown in Table 3.

Close

Fig. 5

Optimization of nprune and degree to accuracy of Mars.

Export to PPT

3.1.3 Optimization of random forest model

Fig. 6 shows the optimization of the RF model by the two hyperparameters ntrees and mtry. The mtry represents the number of variables to randomly sample as candidates at each split. When the value of mtry is larger, it means that the decision tree is more likely to select important features that are related to the outcome variables of most splits. The ntrees represent the number of trees. When the value of mtry increases from 2 to 6, the R² increases significantly. If mtry continues to increase, the accuracy of the model does not change significantly. Increasing the number of trees in the RF model within the first 20 will better fit the model's performance, but further increasing the number of trees from 20 will not significantly change the model's performance. The optimal combination of hyperparameters for the RF model is shown in Table 3.

Close

Fig. 6

Optimization of ntrees and mtry to accuracy of RF.

Export to PPT

3.1.4 Optimization of gradient boosting machine model

Fig. 7 shows the optimization of the GBM Model accuracy for the three hyperparameters shrinkage, ntrees, and interaction depth. As can be seen from Table 2, ntrees is the number of decision trees representing the basic unit of GBM. In Fig. 7, the accuracy of the model is greatly improved when the number of trees increases to 200, and when the number of trees exceeds 200, it has little effect on the accuracy of the model. The shrinkage is considered as a learning rate in the GBM model, which is used for reducing, or shrinking, the impact of each additionally fitted base-learner (decision tree). It reduces the size of incremental steps and thus penalizes the importance of each consecutive iteration. From Fig. 7, it can be seen that with a shrinkage of 0.35, the number of trees required to achieve stability is less than 0.05. Moreover, the interaction depth is the maximum number of nodes in each tree. When the interaction depth is 1, the accuracy of model is the poorest and R² is only 0.90–0.95. When shrinkage is 0.05, the accuracy increases significantly with increasing number of trees and increasing interaction depth. The optimal combination of hyperparameters for the GBM model is shown in Table 3.

Close

Fig. 7

Optimization of three hyperparameters to accuracy of GBM.

Export to PPT

3.1.5 Comparison of accuracy and predictive performance of different machine learning models

Table 3 shows the best accuracy obtained by four optimal hyperparameter combinations of machine learning models, and Fig. 8 presents the comparison of the accuracy of the different machine learning models. Table 3 shows that the GBM model has the highest R² of 0.99 for the following combination of hyperparameters: shrinkage = 0.15, ntrees = 400, and interaction depth = 9. In addition, the time cost of the GBM model is about 35 % lower than that of the RF model. The SVR model is second only to the GBM model, and its R² can also reach 0.98. Of the four machine learning models, the Mars model has the lowest accuracy. When the hyperparameter combination is: degree = 2, nprune = 30, its R² is only 0.95, and RMSE and MAE are 0.29 and 0.20, respectively. But the Mars model is the most time efficient and cost effective of the four models.

Close

Fig. 8

Comparison of accuracy of different machine learning models.

Export to PPT

Fig. 9 is a comparison of the generalization performance of four machine learning models that predict data with experimental data. It can be seen from the figure that most of the data points generated by the GBM model and the SVR model lie on the 45°diagonal. Except for the Mars model and the RF model, the data points generated by other models are very near the center line and essentially situated within a 15 % error region. The generalization ability of the GBM model is also the best among the four machine learning models. The value of R² can reach up to 0.96 but differs from the performance of the model in the training dataset, which shows that the GBM model has overfitting. The generalization performance of the SVR model is second only to the GBM model and the value of R² is 0.95. The generalization performance of the Mars and RF models is poor and their R² is only 0.91. The prediction results are essentially similar to the accuracy results of the model presented in the previous Section 3.1.

Close

Fig. 9

Comparison of predictive performance of different machine learning models.

Export to PPT

Therefore, the SVR model and the GBM model both have good accuracy and generalization performance, which can not only predict well the CO₂ adsorption performance of zeolites synthesized from different solid wastes but also overcome some deficiencies of traditional adsorption isotherm models (e.g., Langmuir model). This is because the machine learning model cannot be constrained by various factors such as the type of raw material and adsorption conditions, and the model can be built directly from actual experimental data without considering some implicit assumptions in traditional models(Meng et al., 2019, Zhu et al., 2020). Thus, the machine learning model constructed in this study is applied to the research of CO₂ adsorption performance of synthetic zeolites for different solid waste, reducing time-consuming and expensive performance testing, which is conducive to the realization of economic and efficient clean production and environmental sustainability.

3.2.1 Variable importance analysis

Fig. 10 illustrates the results of a global sensitivity analysis of nine variables from the four machine learning models using permutation techniques. Fig. 11 shows nine input parameters divided into three types of variables. These include the alkali fusion conditions and hydrothermal conditions during synthesis, and the adsorption conditions during adsorption experiments. The nine variables are divided into three types, alkali fusion conditions: raw material (R), alkali fusion temperature (T₁), and alkali fusion time (t₁); hydrothermal conditions: hydrothermal temperature (T₂), silicon aluminum molar ratio of the synthetic precursor n(Si:Al), and hydrothermal time (t₂); adsorption conditions: adsorption pressure (P), adsorption temperature (T₃), zeolite specific surface area (S). From Fig. 10, it can be seen that the results of the variable importance analysis of these four machine learning models are all different, and it is difficult to explain exactly which variable is closest to the experimental situation and has the greatest influence on the CO₂ adsorption performance of zeolite. Some variable importance tools usually do not take into account the fact that many predictive models can fit the data almost equally well, leading to the Rashomon effect in statistics(Fisher et al., 2019, Del Giudice and Marco, 2021). Furthermore, Fig. 11 shows that the adsorption conditions have the greatest influence on the CO₂ adsorption performance of zeolite, followed by the hydrothermal conditions in the four machine learning models. This is because the zeolites specific surface area (S) is a very important parameter for CO₂ adsorption. These rough and irregular surfaces result from the pore filling of large cations or the partial fusion of cationic compounds during hydrothermal treatment. For many microporous structures, it should be noted that the adsorption capacity depends on the impregnated cations rather than the structure of the adsorbent and that these cations will dominate the adsorption force between the gas molecules and the solid adsorbent through bilateral interactions(Pirngruber et al., 2010, Zukal et al., 2010). At the same time, according to Le Chatelier's principle(Pourhakkak et al., 2021), the adsorption temperature and adsorption pressure change the adsorption performance of zeolite adsorbent with the change of temperature and pressure, so the adsorption amount changes with the change of adsorption temperature and pressure.

Close

Fig. 10

Ranking the importance of different machine learning models on the CO₂ adsorption performance variables of two-step synthesized zeolites.

Export to PPT

Close

Fig. 11

The relative importance of three types of input variables in different machine learning models.

Export to PPT

Fig. 12 shows the values of Shapley in four machine learning models using the local sensitivity analysis method. This value is the average value calculated for the order of arrangement of ten different variables, shown in red and green, respectively, representing negative mean and positive means. The figure also summarizes the distribution of the contribution of each explanatory variable in different rankings represented by purple boxplots. From Fig. 12, it can be concluded that hydrothermal conditions are also one of the important determining factors affecting CO₂ adsorption in the four machine learning models. It is worth noting that the type of raw material is not the factor that determines the adsorption performance, and that the silicon-aluminum molar ratio n(Si:Al) in the precursor may have a greater influence on the adsorption performance of zeolite. This also shows that regardless of the kind of the industrial solid waste, as long as the reaction precursor is rich in silicon aluminum content, the different silicon aluminum content in the hydrothermal will affect the pH and silicon aluminum framework structure of the synthesized zeolite, thus influencing the adsorption performance and selectivity of polar molecules. In addition, the hydrothermal temperature (T₂) is also a very important parameter for determining the structure of zeolites, leading to high solubility of reactants and high values of crystallization rate constants, which cause rapid crystallization to form large crystals(Kotova et al., 2016, Tauanov et al., 2018). At the same time, the activity of water, which serves as a solvent in zeolite synthesis, increases sharply with temperature and promotes the formation of zeolites.

Close

Fig. 12

Shapley values in four machine learning models.

Export to PPT

3.2.2 Effects of variables correlations and interactions

The GBM and SVR models with better model accuracy and generalization ability were selected to analyze the correlations between variables in this study. Fig. 13 shows that the results of the interaction of the GBM and SVR model variables are similar compared to the raw dataset. From the previous section, it is clear that the factor that strongly influences CO₂ adsorption performance is zeolite specific surface area (S). Whether it is alkali fusion conditions or hydrothermal conditions, it has some influence on the specific surface area of the zeolite after crystallization. Among them, the zeolite specific surface area (S) shows a strong positive correlation with the hydrothermal temperature (T₂) and is 0.61 and 0.47 for GBM and SVR, respectively, with an error of only 1.6 % and 21.6 % compared to the raw data. The hydrothermal temperature affects the hydrothermal reaction rate during zeolite synthesis, which in turn affects the autogenous pressure in the reactor and changes the zeolite crystallization products. If the hydrothermal temperature is not high enough, this will result in a low specific surface area for zeolite formation, and the rate of formation of the various crystals and the aspect ratio of the crystals will also be affected by the temperature(Bortolatto et al., 2017, Tauanov et al., 2018, Khajeh Amiri et al., 2019). Therefore, at a suitable hydrothermal temperature and hydrothermal time, zeolite crystals with relatively uniform particle size can be obtained, which affects the specific surface area of the zeolite. In addition, zeolite specific surface area (S) and fusion temperature (T₁) showed a strong negative correlation, which was −0.55 and −0.45 for GBM and SVR, respectively, with errors of 8.3 % and 25 %. This is because within a certain range, too high a melting temperature of the alkalis destroys the surface structure of the zeolite, resulting in a smaller specific surface area. Fig. 13 also shows that the silicon-aluminum molar ratio n(Si:Al) and the fusion time (t₁) also have a strong positive correlation with the zeolite specific surface area. The higher the n(Si:Al), the larger the zeolite specific surface area of framework finally formed, and the acidity gradually increases(Chaves et al., 2015). Fig. 14 shows the comparison of break-down plot with variables interaction of the GBM and SVR models to determine the effect of the interaction between different variables on the adsorption performance of zeolite for carbon dioxide. The interaction between each variable is included as a single bar in the figure. Since the influence of each variable cannot be disentangled, the graph uses just that single bar to represent the contribution of each variable. It can be seen from the Fig. 14 (a) that compared with the average model prediction, the specific surface area of 79.2 reduces the performance of zeolite for carbon dioxide, and the adsorption performance is further reduced at a silicon-aluminum ratio of 3.63. However, since the adsorption temperature is 298, this increases the adsorption performance of the zeolite.

Close

Fig. 13

Comparison of variable correlation heatmap.

Export to PPT

Close

Fig. 14

Comparison of break-down plot with variable interaction.

Export to PPT

In both the global and local sensitivity analyses, alkali fusion conditions were found to be the weakest variable affecting zeolite adsorption performance. In the GBM model with the best generalization performance, the importance of alkali fusion time (t₁) is also at the bottom. However, a large number of publications indicate that the step of alkali fusion can decompose the Silicon and aluminum rich crystalline phases and form soluble aluminates and silicates, making them highly reactive and promoting the formation of zeolites(Ayele et al., 2016, Jin et al., 2021, Lin and Chen, 2021). This is most likely due to the fact that the machine learning model is limited by the original data sources of 11 literature. The limited amount of data under alkali fusion conditions caused the model to learn incorrectly, so the final result did not match the actual situation. In addition, Fig. 13 shows an unusually strong correlation between the alkali fusion time (t₁) and the type of raw material (R), again due to the small amount of data on alkali fusion conditions for each raw material in the data set. Therefore, this requires a larger amount of experimental data. In the future, the influence of alkali fusion conditions on the adsorption performance of zeolite will be further researched and evaluated to provide guidance for the cleaner process of industrial solid waste and reduce the experimental time and cost.