Analyzing the Impact of Feature Selection on Crop Yield Prediction ()
1. Introduction
Recently, data has been rapidly increasing in both the number of samples and features. To determine data trends, it is necessary to study how to utilize these large-scale data. The huge number of high dimensional data has a big challenge due to the presence of noisy, redundant, and irrelevant data as it increases the computational complexity of the model and overcomes overfitting problem. So, feature selection is necessary as it allows us to choose relevant features from the original data by removing noisy, irrelevant, and redundant features.
In machine learning problems, high dimensional data, especially in terms of many features, is increasing these days [1]. In data analysis applications, the dataset may have many features which determine the applicability and usability of the data [2]. The reduction of the large number of features to smaller ones with preserving the relevant information while discarding the redundant features is referred to as feature selection [3]. Nowadays agriculture data is becoming high dimensional data, especially in terms of features. So, we must determine relevant features from this high-dimensional data. We can choose correlated and non-redundant features, so feature selection plays an important role.
Feature selection process involves the following steps.
Understand the Problem: Gain insights into the problem domain, including the factors that influence crop yield such as weather conditions, soil characteristics, crop type, farming practices, etc.
Data Collection: Gather a comprehensive dataset containing various features related to crop yield, including historical yield data, weather data, soil data, crop management practices, etc.
Exploratory Data Analysis: Perform Exploratory Data Analysis to understand the relationships between different features and the target variable. This involves visualizing data, computing summary statistics, and identifying correlations.
Agriculture plays an impactful role in Indian economy and crop yield has a major contribution. This section includes details about different methods for crop yield prediction. Feature selection techniques are classified into three major types called filter, wrapper, and Embedded approaches [4] [5]. The wrapper method is used for selecting features, but the approach is computationally expensive and does not provide optimal number features. Filter approach is independent of any learning algorithm [6]. The embedded approach uses the advantage of both supervised and unsupervised approaches. By combining feature selection techniques, we can improve the learning performance [7]. Forward selection, wrapper technique is used for selecting important features [8], correlation-based filter feature selection algorithm is used in [9], random forest variable Importance embedded technique is used in [10]. These algorithms do not result in significant features. To overcome this drawback a hybrid approach is used which is combination of Correlation and Sequential Forward Feature Selection [11]. Maya Gopal and Bhargavi [12] analyzed the performance of machine learning algorithms using various feature selection techniques for crop yield prediction. Maya Gopal and Bhargavi [13] proposed sequential forward selection technique to find the “optimal” feature subset by iteratively selecting features based on the performance of the classifiers. Zahra Karimi et al. [14], implemented hybrid filter feature selection scheme for intrusion detection. The experimental results show that the proposed technique offers higher accuracy than other methods. Surabhi Chouhan et al. [15], proposed a method of applying the Particle Swarm Optimization and Support Vector Machine (PSO-SVM) to select features from a dataset. Akhiat, Y [16] proposed an ensemble selection framework to improve the proposed feature selection algorithm to reduce the overfitting problem using multi-bagging and dropout technique. According to G. Mariammal et al. [17] a modified recursive feature elimination approach was presented for selecting acceptable features for crop prediction using bagging methodologies. A. Suruliandi et al. [18], carried out comparative examination of multiple wrapper feature selection methodologies to recommend the best suited crop. It is found that fusion of Recursive Feature Elimination approach and Adaptive Bagging classifier performed well as compared to remaining methods. Alberto Gonzalez-Sanchez et al. [19], evaluated different model correctness and used artificial neural networks to demonstrate result. Hsu HH, Hsieh CW, Da LuM [20] used hybrid feature selection technique which is combination of filters and wrappers. Dewi C [21], used random forest and support vector machine techniques to improve the model accuracy using feature selection. Nicholas Pudjihartono [22] reviewed different feature selection techniques for disease risk prediction. M.S. Suresh Sumi [23], used combination of Filter and Wrapper feature selection techniques to improve classification Accuracy. S. P. Raja [24] applied different feature selection techniques and used ensembled model to improve the performance. Thomas. Rincy [25], Kajal P Parmar [26], Younes Bouchlaghem [27] performed a detail review of different feature selection techniques that can be used for effective accuracy of the machine learning model. Mrs. K.A. Poornima [28], combined Correlation and Sequential Forward Feature Selection techniques to improve accuracy in crop prediction.
The Efficient method for prediction of crop yield is an important process of agriculture and machine Learning is playing a major role in this area in recent years. Feature selection and prediction are critical machine learning techniques, and it improves prediction accuracy and helps to avoid processing of unwanted feature data and minimizing execution time. For selecting relevant features in literature mainly filter, wrapper, and embedded methods are used frequently. To improve the accuracy of prediction efficient feature selection techniques should be introduced. As a part of research this study implemented hybrid feature selection technique namely Rank-based W2FE feature selection techniques which is a fusion of the efficient techniques from Filter, Wrapper and Embedded method. The Wrapper 2 Filter Embedded (W2FE) method is a combination of two wrapper methods, one Filter and one embedded method. The result of W2FE feature selection model has been given to rank-based and weighted rank-based algorithm to determine the most suitable features for crop yield prediction. Weighted rank-based W2FE feature selection algorithm gives better results as compared to the Rank-based W2FE feature selection techniques. The main contributions of this research are summarized as follows. In this research work various features affect crop yield. has been studied. Different feature selection techniques have been studied and implemented, to determine relevant features for predicting crop yield. In this study Linear regression, Random Forest, and Decision Tree regressor methods are used for assessing the results given by different feature selection techniques. The most suitable methods are chosen after evaluating performance using different machine learning models. hybrid feature selection techniques, namely rank-based W2FE feature selection algorithm and Weighted rank-based W2FE feature selection algorithm have been proposed and implemented by integrating efficient feature selection techniques to select optimal feature sets and hence improves efficiency of crop yield prediction.
2. Method
In this work, different feature selection techniques have been implemented and performance of different feature selection techniques has been analyzed for prediction of the crop yield. Hybrid feature selection techniques have been implemented by combining suitable methods from different feature selection techniques to select a minimal set of features hence improved crop yield prediction. This study considered metrological data of Marathwada region from India country. Weather data is collected from the Meteorological Department Pune, crop production data has been collected from Government Website.
2.1. Study Area
This study area has been used as Marathwada region from Maharashtra state in India. The Marathwada have different principal foods like Soybean, Wheat, Jawar etc. This study considered 8 districts from the Marathwada region.
2.2. Dataset Details
This study has used crop production Dataset used from publicly available records of the Indian Government for the years 2001 to 2019 and the weather features details have been sourced from Indian Meteorological Department in Pune. Features considered for this study are rainfall details, temperature details, precipitation, evaporation, and area.
2.3. Methodology
1) Understand the Problem: Gain insights into the problem domain, including the factors that influence crop yield such as weather conditions, soil characteristics, crop type, farming practices, etc.
2) Data Collection: Gather a comprehensive dataset containing various features related to crop yield, including historical yield data, weather data, soil data, crop management practices, etc.
3) Exploratory Data Analysis: Perform Exploratory Data Analysis to understand the relationships between different features and the target variable (crop yield). This involves visualizing data, computing summary statistics, and identifying correlations.
4) Feature Selection Techniques: Apply different feature selection techniques and analyze the performance.
2.4. Proposed Framework
In the proposed work, a hybrid method-W2FE Rank-based/Weighted Rank-based Feature selection technique for feature selection has been proposed which consists of fusion of different feature selection techniques. The architecture of the proposed model is represented in Figure 1. Different feature selection techniques have been implemented and performance has been evaluated.
Figure 1. The architecture of the proposed feature selection model.
Architecture of proposed hybrid feature selection technique model consists of following major components:
1) Data Preprocessing component: In data pre-processing multicollinearity among the data has been removed and data has been standardized by using Min-Max normalization technique.
2). Feature selection component: Preprocessed data has been given as a input to the feature selection module. In this module, the most suitable methods from different types of feature selection technique is being used, and the best features are identified for crop yield prediction. Rank-based feature elimination technique is used to rank features while determining relevant features for crop yield prediction.
2.5. Implementation Details
As a part of the research, this study implemented hybrid model: W2FE Rank-based/Weighted Rank-based Feature selection algorithm which is a fusion of the most suitable techniques from Filter, Wrapper and Embedded method. The result of hybrid model has been given to rank-based and weighted rank-based algorithm to determine the most suitable features for crop yield prediction rank-based feature selection algorithm, weighted rank-based feature.
Algorithmic Details:
1) W2FE Rank-based /Weighted Rank-based Feature Selection Algorithm:
Input: feature set F = {ƒ1 › ƒ2 › ··· › fm} //m is the number of features
Output: optimal feature set R
a) Create a Feature Score Table (FST) of 8 rows and 7 columns, having the evaluation scores provided by the different methods for feature selection of each feature.
b) F1 = Feature set after applying Filter based technique.
c) F2 = Feature set after applying Wrapper based technique.
d) F3 = Feature set after applying Embedded technique.
e) F = F1UF2UF3.
f) Apply Rank-based and Weighted Rank-based algorithm.
g) Calculate the total score by taking sum of all scores for every feature.
h) If score >0.5* number of features, then.
i) Add feature in the selected features.
2) Feature Selection Techniques:
a) Filter Methods: Uses statistical measures like correlation, chi-square, or mutual information to rank features based on their relevance to the target variable.
b) Wrapper Methods: Employ algorithms like Recursive Feature Elimination (RFE) or Forward/Backward Selection, which select subsets of features based on the performance of a specific machine learning algorithm.
c) Embedded Methods: Some machine learning algorithms inherently perform feature selection during training. Examples include LASSO (L1 regularization) and tree-based methods like Random Forests.
3) Rank-Based Feature Selection Algorithm:
Input: feature set F = {ƒ1 › ƒ2 › ··· › fm} //m is the number of features
Output: ranked features R
Create a Feature Score Table (FST) of 8 rows and 12 columns, having the evaluation scores provided by the different methods for feature selection of each feature.
If feature has been selected, then provide score 1 otherwise 0
Calculate Total Score by taking sum of all scores for every feature.
If score >0.5* number of features, then
Add feature in the selected features.
4) Weighted Rank-Based Feature Selection Algorithm:
Input: feature set F = {ƒ1 › ƒ2 › ··· › fm} //m is the number of features
Output: ranked features R
Create a Feature Score Table (FST) of 8 rows and 7 columns, having the evaluation scores provided by the different methods for feature selection of each feature.
Create rank table from FST.
For every selected feature assign weight depending on rank of feature.
Calculate the total score by taking sum of all scores for every feature.
If score >0.5* number of features, then.
Add feature in the selected features.
3. Results and Discussion
This research work analyses different types of feature selection techniques like Filter feature selection technique, Wrapper feature selection technique and Embedded feature selection techniques, Rank-based feature selection technique, Weighted feature selection technique and Hybrid feature section techniques: W2FE Rank-Based/Weighted Rank-Based Feature selection algorithm.
1) Analysis of Basic Feature Selection Techniques: The feature subsets returned by different basic feature selection techniques are shown in Table 1.
Table 1 depicts the result of different rank-based Feature selection Algorithm implemented as a part of said research.
Table 1. Feature selection algorithm result.
| Feature Selection Method/Feature |
MMAX |
HMAX |
MMIN |
LMIN |
TRMF |
HVRYF |
MWS |
AREA |
RD |
MEVP |
MWS |
PRCP |
HMD |
| Correlation |
N |
N |
N |
N |
Y |
Y |
N |
Y |
Y |
N |
N |
N |
N |
| Chi-Square |
N |
N |
Y |
N |
Y |
N |
N |
N |
N |
Y |
N |
Y |
N |
Forward
Elimination Technique |
N |
N |
N |
Y |
N |
Y |
Y |
N |
N |
Y |
Y |
N |
N |
Backward
Elimination
Selection |
N |
N |
N |
Y |
Y |
N |
Y |
N |
N |
Y |
Y |
N |
N |
SFS Linear
Regression |
N |
Y |
Y |
N |
N |
Y |
N |
N |
N |
Y |
N |
N |
N |
| SFS KNN |
Y |
Y |
N |
Y |
N |
N |
N |
N |
Y |
N |
N |
N |
N |
| RFE |
N |
N |
Y |
Y |
N |
N |
N |
N |
N |
N |
N |
Y |
Y |
| Random Forest Importance |
Y |
N |
N |
N |
Y |
Y |
N |
Y |
N |
N |
N |
N |
N |
| Embedded LGB |
Y |
Y |
N |
Y |
N |
N |
N |
N |
N |
N |
Y |
N |
N |
2) Rank-based Feature Selection Technique: After applying different feature selection techniques, the Rank-based feature selection technique has been applied on the features returned by different feature selection techniques. Rank-based feature selection technique returns features whose score is minimum 0.5.
Table 2 shows the score of every feature after applying different feature selection techniques as a part of said research.
Table 2. Rank-based feature selection algorithm.
Feature
Selection Method/Feature |
MMAX |
HMAX |
MMIN |
LMIN |
TRMF |
HVRYF |
MWS |
AREA |
RD |
MEVP |
MWS |
PRCP |
HMD |
| Correlation |
N |
N |
N |
N |
Y |
Y |
N |
Y |
Y |
N |
N |
N |
N |
| Chi-Square |
N |
N |
Y |
N |
Y |
N |
N |
N |
N |
Y |
N |
Y |
N |
Forward
Elimination Technique |
N |
N |
N |
Y |
N |
Y |
Y |
N |
N |
Y |
Y |
N |
N |
| Backward Elimination Selection |
N |
N |
N |
Y |
Y |
N |
Y |
N |
N |
Y |
Y |
N |
N |
SFS Linear
Regression |
N |
Y |
Y |
N |
N |
Y |
N |
N |
N |
|
N |
N |
N |
| SFS KNN |
Y |
Y |
N |
Y |
N |
N |
N |
N |
Y |
N |
N |
N |
N |
| RFE |
N |
N |
Y |
Y |
N |
N |
N |
N |
N |
N |
N |
Y |
Y |
| Random Forest Importance |
Y |
N |
N |
N |
Y |
Y |
N |
Y |
N |
N |
N |
N |
N |
| Embedded LGB |
Y |
Y |
N |
Y |
N |
N |
N |
N |
N |
N |
Y |
N |
N |
| Total Score |
4 |
3 |
2 |
5 |
4 |
4 |
2 |
2 |
2 |
3 |
3 |
2 |
1 |
After applying Rank-based technique on the above result it has returned [MMAX, HMAX, TRMF, HVRF] features.
3) Weighted Rank-based feature selection technique: For improving performance Weighted Rank-based feature selection technique has been implemented. In weighted Rank-based selection technique weights are assigned to every feature depending on the importance given by different feature selection technique. Table 3 shows the score of every feature after applying Weighted Rank-based feature selection technique returns features whose weighted score is minimum 0.5.
The optimal features selected by rank-based technique are “MMAX”, “HMAX”, “LMIN”, “HVRF” features.
4) Performance Evaluation of Feature Selection Techniques using Machine Learning Models:
The performance of these techniques has been evaluated using different machine learning models like Linear Regressor, Decision Tree and Random Forest. Table 4 shows the performance of different feature selection techniques.
After applying different feature selection techniques, it is found that Embedded LGB (E), SFS LR (W), Chi-Square (F), FST/BET (W) performed better as compared to other feature selection techniques.
5) Performance Evaluation of W2FE Rank-Based and Weighted Rank-Based Feature Selection Techniques using Machine Learning Models:
Table 3. Weighted rank-based feature selection algorithm.
Feature
Selection
Method/Feature |
MMAX |
HMAX |
MMIN |
LMIN |
TRMF |
HVRYF |
MWS |
AREA |
RD |
MEVP |
MWS |
PRCP |
HMD |
| Correlation |
N |
N |
N |
N |
Y(3) |
Y(4) |
N |
Y(2) |
Y(2) |
N |
N |
N |
N |
| Chi -Square |
N |
N |
Y |
N |
Y |
N |
N |
N |
N |
Y |
N |
Y(1) |
N |
Forward
Elimination Technique |
N |
N |
N |
Y(2) |
N |
Y(4) |
N |
N |
N |
Y(1) |
Y(3) |
N |
N |
Backward
Elimination Selection |
N |
N |
N |
Y(2) |
N |
Y(4) |
N |
N |
N |
Y(1) |
Y(3) |
N |
N |
SFS Linear
Regression |
N |
Y(4) |
Y(3) |
N |
N |
Y(2) |
N |
N |
N |
Y(1) |
N |
N |
N |
| SFS KNN |
Y(4) |
Y(3) |
N |
Y(2) |
N |
N |
N |
N |
Y(1) |
N |
N |
N |
N |
| RFE |
N |
N |
Y(4) |
Y(3) |
N |
N |
N |
N |
N |
N |
N |
Y(2) |
Y(1) |
| Random Forest Importance |
Y(4) |
N |
N |
N |
Y(3) |
Y(2) |
N |
Y(1) |
N |
N |
N |
N |
N |
| Embedded LGB |
Y(4) |
Y(3) |
N |
Y(2) |
N |
N |
N |
N |
N |
N |
Y(1) |
N |
N |
| Total Score |
12 |
10 |
7 |
9 |
6 |
12 |
2 |
2 |
2 |
3 |
7 |
3 |
1 |
Table 4. Performance evaluation of different feature selection techniques.
| Feature Selection Technique |
Features Selected |
ML Algorithm |
MAE |
MSE |
RMSE |
| FST/BET (W) |
HVYRF, “MWS”, “LMIN”, “MEVP” |
LR |
2.38 |
5.87 |
0.21 |
| DT |
0.21 |
0.05 |
0.21 |
| RF |
0.23 |
0.06 |
0.25 |
| Co-relation (F) |
“HVYRF”, “TMRF”, “Area”, “RD” |
LR |
0.95 |
1.17 |
1.08 |
| DT |
0.45 |
0.37 |
0.61 |
| RF |
0.27 |
0.11 |
0.33 |
| Chi-Square (F) |
“MMIN”, “TMRF”, “MEVP”, “PRCP” |
LR |
0.21 |
0.06 |
0.24 |
| DT |
0.33 |
0.16 |
0.41 |
| RF |
0.22 |
0.08 |
0.28 |
| SFS LR (W) |
“HMAX”, “MMIN”, “HVYRF”, “MEVP” |
LR |
0.95 |
0.93 |
0.97 |
| DT |
0.13 |
0.02 |
0.16 |
| RF |
0.14 |
0.03 |
0.18 |
| SFS KNN (W) |
“MMAX”, “HMAX”, “LMIN”, “RD” |
LR |
0.43 |
0.24 |
0.49 |
| DT |
0.29 |
0.12 |
0.34 |
| RF |
0.3 |
0.12 |
0.32 |
| RFE (W) |
“MMIN”, “LMIN”, “PRCP”, “HMD” |
LR |
0.61 |
0.44 |
0.67 |
|
|
DT |
0.45 |
0.23 |
0.48 |
| RF |
0.46 |
0.22 |
0.47 |
| RFI (E) |
“MMAX”, “TMRF”, “HVYRF”, “AREA” |
LR |
0.62 |
0.41 |
0.64 |
| DT |
0.43 |
0.39 |
0.62 |
| RF |
0.31 |
0.12 |
0.34 |
| Embedded LGB (E) |
“MMAX”, “HMAX”, “MMIN”, “LMIN” |
LR |
0.18 |
0.05 |
0.23 |
| DT |
0.31 |
0.23 |
0.48 |
| RF |
0.27 |
0.1 |
0.32 |
| Rank-based Technique |
MMAX, HMAX, TRMF, HVRF |
LR |
0.3 |
0.15 |
0.38 |
| DT |
0.66 |
0.5 |
0.71 |
| Weighted Rank Based Technique |
“MMAX”, “HMAX”, “LMIN”, “HVRF” |
RF |
0.35 |
0.13 |
0.36 |
| LR |
0.45 |
0.25 |
0.5 |
| DT |
0.45 |
0.25 |
0.53 |
| RF |
0.42 |
0.25 |
0.5 |
Table 5. (a). W2FE rank-based feature selection techniques. (b). W2FE weighted rank-based feature selection techniques.
| (a) |
Feature Selection
Technique |
MMAX |
HMAX |
MMIN |
LMIN |
HVYRF |
TMRF |
MEVP |
MWS |
PRCP |
| Embedded LGB (E) |
Y |
Y |
Y |
Y |
- |
- |
- |
- |
- |
| SFS LR (W) |
- |
Y |
Y |
- |
Y |
- |
Y |
- |
- |
| Chi-Square (F) |
- |
- |
Y |
- |
- |
Y |
Y |
- |
Y |
| FST/BET (W) |
- |
- |
- |
Y |
Y |
- |
Y |
Y |
- |
| Total Rank |
1 |
2 |
3 |
2 |
2 |
1 |
3 |
1 |
1 |
| (b) |
Feature Selection
Technique |
MMAX |
HMAX |
MMIN |
LMIN |
HVYRF |
TMRF |
MEVP |
MWS |
PRCP |
| Embedded LGB (E) |
Y(4) |
Y(3) |
Y(2) |
Y(1) |
- |
- |
- |
- |
- |
| SFS LR (W) |
- |
Y(4) |
Y(3) |
- |
Y(2) |
- |
Y(1) |
- |
- |
| Chi-Square (F) |
- |
- |
Y(4) |
- |
- |
Y(3) |
Y(2) |
- |
Y(1) |
| FST/BET (W) |
- |
- |
- |
Y(2) |
Y(4) |
- |
Y(1) |
Y(3) |
- |
| Total Weighted Rank |
4 |
7 |
9 |
3 |
6 |
3 |
4 |
3 |
1 |
Features selected by Embedded LGB (E), SFS LR (W), Chi-Square (F), FST/BET (W) methods have been integrated and Rank-based and Weighted Rank-based techniques have been applied on the given data. Table 5(a) and Table 5(b) show the result of W2FE rank-based feature selection technique and W2FE Weighted Rank-based Feature Selection Technique.
The Rank-based Selective Feature Selection Techniques returns HMAX, MMIN, LMIN, HVRF, MEVP as optimal feature set.
The Weighted Rank-based Selective Feature Selection Techniques returns MMIN, HMAX, HVYRF as optimal feature set.
Table 6. Performance evaluation of W2FE rank-based and weighted rank-based feature selection techniques.
| Feature Selection Technique |
Features Selected |
ML Techniques |
MAE |
MSE |
RMSE |
W2FE Rank-Based Feature
Selection Technique |
HMAX, MMIN, LMIN HVRF, MEVP |
LR |
1.15 |
1.5 |
1.22 |
| DT |
0.23 |
0.12 |
0.34 |
| RF |
0.31 |
0.12 |
0.35 |
| W2FE Weighted Rank-Based Feature Selection Technique |
HMAX, MMIN, HVRF |
LR |
0.3 |
0.15 |
0.38 |
| DT |
0.22 |
0.08 |
0.29 |
| RF |
0.18 |
0.05 |
0.22 |
Table 6 shows the performance evaluation of W2FE Rank-based and Weighted Rank-based Feature Selection Techniques.
After evaluation these techniques it is found that W2FE Weighted Rank-based Feature technique performed well as compared to the W2FE Rank-based Selection Technique.
4. Conclusion
In this paper, different types of feature selection techniques have been implemented and performance of those techniques have been analyzed by using different machine learning models. This research work used a two-stage feature selection approach which is integration of efficient feature selection techniques which has been found after analyzing performance of basic feature selection techniques and rank-based techniques. In W2FE Rank-based Feature Selection Technique and Weighted Rank-based Feature Selection Techniques, a filter, wrapper and embedded methods are used to determine features which are mostly selected by different feature selection techniques. These feature candidate sets are further pruned by using rank-based techniques. It is found that by exploiting strengths of the different methods and creating fusion of different methods allow better accuracy and stability than relying on any single feature selection method. The paper analyzed performance of two feature elimination method namely W2FE Weighted Rank-based Selective Feature Selection Techniques and W2FE Rank-based Feature Selection Techniques and found that Weighted Rank-based Selective Feature Selection Techniques gives better results as compared to Rank-based Selective Feature Selection Techniques.