There are several commonly used evaluation criteria for feature selection. These criteria help to assess the quality and relevance of features in a dataset. Some of the widely used evaluation criteria are:

  1. Mutual Information: Mutual information measures the dependency between two variables. In feature selection, it is used to estimate the relevance of each feature with the target variable.

  2. Chi-square (χ²) test: Chi-square test is used to determine the independence between two categorical variables. It is commonly used in feature selection to assess the relevance of categorical features with the target variable.

  3. Correlation coefficient: Correlation coefficient measures the linear relationship between two variables. It is commonly used in feature selection to identify features that are highly correlated with the target variable.

  4. Information gain: Information gain is a measure used in decision trees and is calculated based on the entropy. It determines the extent to which the presence of a feature reduces the uncertainty about the target variable.

  5. Recursive Feature Elimination (RFE): RFE is an iterative feature selection method that works by recursively eliminating features and building a model on the remaining features. The effectiveness of each feature is assessed by the model's performance.

  6. Regularization: Regularization techniques like L1 and L2 regularization can be used to penalize the coefficients of less significant features. By minimizing the penalty term, less relevant features are suppressed.

  7. Wrapper methods: Wrapper methods evaluate subsets of features by training and testing a model on them. Performance measures, such as accuracy, are then used to select the best subset of features.

  8. Embedded methods: Embedded methods incorporate feature selection within the model-building process. By considering feature importance during model training, these methods select the most relevant features.

  It is worth mentioning that the choice of evaluation criteria depends on the nature of the data and the specific problem at hand. Different evaluation criteria may yield different results, so it is important to carefully choose the most appropriate criteria for feature selection.