Python implementations of the Boruta all-relevant feature selection method.

Overview

boruta_py

License PyPI version Anaconda-Server Badge

This project hosts Python implementations of the Boruta all-relevant feature selection method.

Related blog post

How to install

Install with pip:

pip install Boruta

or with conda:

conda install -c conda-forge boruta_py

Dependencies

  • numpy
  • scipy
  • scikit-learn

How to use

Download, import and do as you would with any other scikit-learn method:

  • fit(X, y)
  • transform(X)
  • fit_transform(X, y)

Description

Python implementations of the Boruta R package.

This implementation tries to mimic the scikit-learn interface, so use fit, transform or fit_transform, to run the feature selection.

For more, see the docs of these functions, and the examples below.

Original code and method by: Miron B. Kursa, https://notabug.org/mbq/Boruta/wiki/FAQ

Boruta is an all relevant feature selection method, while most other are minimal optimal; this means it tries to find all features carrying information usable for prediction, rather than finding a possibly compact subset of features on which some classifier has a minimal error.

Why bother with all relevant feature selection? When you try to understand the phenomenon that made your data, you should care about all factors that contribute to it, not just the bluntest signs of it in context of your methodology (yes, minimal optimal set of features by definition depends on your classifier choice).

What's different in BorutaPy?

It is the original R package recoded in Python with a few added extra features. Some improvements include:

  • Faster run times, thanks to scikit-learn

  • Scikit-learn like interface

  • Compatible with any ensemble method from scikit-learn

  • Automatic n_estimator selection

  • Ranking of features

  • Feature importances are derived from Gini impurity instead of RandomForest R package's MDA.

For more details, please check the top of the docstring.

We highly recommend using pruned trees with a depth between 3-7.

Also, after playing around a lot with the original code I identified a few areas where the core algorithm could be improved/altered to make it less strict and more applicable to biological data, where the Bonferroni correction might be overly harsh.

Percentile as threshold
The original method uses the maximum of the shadow features as a threshold in deciding which real feature is doing better than the shadow ones. This could be overly harsh.

To control this, I added the perc parameter, which sets the percentile of the shadow features' importances, the algorithm uses as the threshold. The default of 100 which is equivalent to taking the maximum as the R version of Boruta does, but it could be relaxed. Note, since this is the percentile, it changes with the size of the dataset. With several thousands of features it isn't as stringent as with a few dozens at the end of a Boruta run.

Two step correction for multiple testing
The correction for multiple testing was relaxed by making it a two step process, rather than a harsh one step Bonferroni correction.

We need to correct firstly because in each iteration we test a number of features against the null hypothesis (does a feature perform better than expected by random). For this the Bonferroni correction is used in the original code which is known to be too stringent in such scenarios (at least for biological data), and also the original code corrects for n features, even if we are in the 50th iteration where we only have k<<n features left. For this reason the first step of correction is the widely used Benjamini Hochberg FDR.

Following that however we also need to account for the fact that we have been testing the same features over and over again in each iteration with the same test. For this scenario the Bonferroni is perfect, so it is applied by deviding the p-value threshold with the current iteration index.

If this two step correction is not required, the two_step parameter has to be set to False, then (with perc=100) BorutaPy behaves exactly as the R version.

Parameters

estimator : object

A supervised learning estimator, with a 'fit' method that returns the feature_importances_ attribute. Important features must correspond to high absolute values in the feature_importances_.

n_estimators : int or string, default = 1000

If int sets the number of estimators in the chosen ensemble method. If 'auto' this is determined automatically based on the size of the dataset. The other parameters of the used estimators need to be set with initialisation.

perc : int, default = 100

Instead of the max we use the percentile defined by the user, to pick our threshold for comparison between shadow and real features. The max tend to be too stringent. This provides a finer control over this. The lower perc is the more false positives will be picked as relevant but also the less relevant features will be left out. The usual trade-off. The default is essentially the vanilla Boruta corresponding to the max.

alpha : float, default = 0.05

Level at which the corrected p-values will get rejected in both correction steps.

two_step : Boolean, default = True

If you want to use the original implementation of Boruta with Bonferroni correction only set this to False.

max_iter : int, default = 100

The number of maximum iterations to perform.

verbose : int, default=0

Controls verbosity of output.

Attributes

n_features_ : int

The number of selected features.

support_ : array of shape [n_features]

The mask of selected features - only confirmed ones are True.

support_weak_ : array of shape [n_features]

The mask of selected tentative features, which haven't gained enough support during the max_iter number of iterations..

ranking_ : array of shape [n_features]

The feature ranking, such that ranking_[i] corresponds to the ranking position of the i-th feature. Selected (i.e., estimated best) features are assigned rank 1 and tentative features are assigned rank 2.

Examples

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from boruta import BorutaPy

# load X and y
# NOTE BorutaPy accepts numpy arrays only, hence the .values attribute
X = pd.read_csv('examples/test_X.csv', index_col=0).values
y = pd.read_csv('examples/test_y.csv', header=None, index_col=0).values
y = y.ravel()

# define random forest classifier, with utilising all cores and
# sampling in proportion to y labels
rf = RandomForestClassifier(n_jobs=-1, class_weight='balanced', max_depth=5)

# define Boruta feature selection method
feat_selector = BorutaPy(rf, n_estimators='auto', verbose=2, random_state=1)

# find all relevant features - 5 features should be selected
feat_selector.fit(X, y)

# check selected features - first 5 features are selected
feat_selector.support_

# check ranking of features
feat_selector.ranking_

# call transform() on X to filter it down to selected features
X_filtered = feat_selector.transform(X)

References

  1. Kursa M., Rudnicki W., "Feature Selection with the Boruta Package" Journal of Statistical Software, Vol. 36, Issue 11, Sep 2010
Comments
  • PR for packaging, Python3 Support, and Issues #3 and #4

    PR for packaging, Python3 Support, and Issues #3 and #4

    First of all, I apologize for writing such a big PR all at once. I was struggling to get Boruta functioning on my problem and these were the things I needed to fix to get it working for me.

    1. PR supports Python 3.5
    2. PR refactored code to support setup.py. Boruta can be uploaded to PyPI or installed with 'python setup.py install'
    3. PR resolves issue # 3 Re slicing Pandas DataFrames (likely only in versions > .17.x?)
    4. PR resolves issue #4 Re all features selected.
    5. I was unsure what to do with the two versions of boruta.py. Patching two files in parallel isn't ideal. The second version seems desirable/valuable so I removed the first from the package.

    I hope you find this useful. Happy to make any changes you'd like. Thanks for implementing Boruta in python!

    opened by mbernico 13
  • catboost-plot-pimp-shapimp

    catboost-plot-pimp-shapimp

    Modifications are:

    • catboost checks (doesn't allow to change random seed after fitting), cases for catboost estimator
    • plot: boxplot of the var. imp history (confirmed, tentative and rejected are colour-coded)
    • categorical feature for boosting, plot (mimicking more or less the R one), OHE is known to lead to deep and unstable trees. LightGBM and CatBoost have native methods to handle cat. pred, this requires to declare which columns are cat (lightgbm has an auto mode but the columns should be integer encoded and dtype set to category)
    • permutation importance and SHAP importance, (the impurity based var.imp being biased towards large card. and numerical predictors and computed on the train set)
    • testing: modif_tests.py for classification and regression testing (including a random cat. pred with large cardinality)

    thanks KR

    opened by ThomasBury 11
  • Make boruta_py suitable for GridSearches

    Make boruta_py suitable for GridSearches

    When using boruta_py in a sklearn gridsearch, the error object has no attribute 'get_params' occurs. It would be interesting, if one could also optimize the parameter of the boruta feature selection

    opened by MaxBenChrist 11
  • Question: Feature Selection for Regression Problems

    Question: Feature Selection for Regression Problems

    The examples provided apply Boruta for feature selection in classification problems. Can Boruta be accurately applied for feature selection in regression problems? If so, what regression estimator would be most appropriate? (i.e. RandomForestRegressor, GradientBoostingRegressor, etc.)

    opened by SymbolicSquared 8
  • XGBoost Support

    XGBoost Support

    Is BorutaPy compatible with XGBoost? If not, would you be interested in a PR for that compatibility (assuming it's possible and I can figure it out)?

    It seems to me that this is not currently supported since I got an error when I tried it with XGBClassifier, but I wanted to know if there's any official word.

    Thanks!

    opened by gaw89 8
  • iteration over a 0-d array in `_nanrankdata`

    iteration over a 0-d array in `_nanrankdata`

    I gather that others have hit this (https://github.com/danielhomola/boruta_py/issues/12) but it still seems like a live issue, I'm afraid. It's hitting when X and y are ndarrays of what looks like the right shape.

    There's a reproducible example on Iris data here.

    opened by nerdcha 8
  • Fixed

    Fixed "Tuple Index Out of range error", unit test and example notebook

    This PR relates to #47, a bug I made in #46. We should only check the size of 1st dimension of not_selected array, as it would be 0 already if all features are relevant.

    This time I have also double-checked that unit test case is passed (fixed a small issue inside as well).

    Here is the output log of unit test:

    $ python unit_tests.py
    ..
    ----------------------------------------------------------------------
    Ran 2 tests in 11.859s
    
    OK
    

    I also discovered some compatibility issues to Python (I'm using 3.6.5) and Pandas in the example notebook while doing correctness test, and it should work well with current version as well!

    opened by guitarmind 7
  • d:\Anaconda3\lib\site-packages\boruta\boruta_py.py:418: RuntimeWarning: invalid value encountered in greater   hits = np.where(cur_imp[0] > imp_sha_max)[0]

    d:\Anaconda3\lib\site-packages\boruta\boruta_py.py:418: RuntimeWarning: invalid value encountered in greater hits = np.where(cur_imp[0] > imp_sha_max)[0]

    d:\Anaconda3\lib\site-packages\boruta\boruta_py.py:418: RuntimeWarning: invalid value encountered in greater hits = np.where(cur_imp[0] > imp_sha_max)[0]

    opened by shushan2017 7
  • iteration over a 0-d array

    iteration over a 0-d array

    Hi Team - I have faced this "iteration over a 0-d array" for a specific data set and read all the QA and understood it is fixed ( if i am right ). But it seems problem persists for A dataset(wine) . There is NO nan values in any rows/columns or full array of nan values.but i am facing this issue.

    It would of great help if u guide me on this , unless i am wrongly coded. Thanks Here is the dataset and code wine.csv.zip FEATURE_SELECTION_BORUTA.py.zip

    opened by Leninstark 5
  • fixed setup.py requirements

    fixed setup.py requirements

    Just had to reinstall, noticed setup.py hard coded to some older versions of scipy, numpy, sklearn.

    This PR updates setup.py such that it requires at least those versions, rather than exactly those versions.

    opened by mbernico 5
  • return importance history from fit

    return importance history from fit

    Addresses issue #87. You already collect the importance histories. Simply adding it to the selector object so that it will be available after calling fit.

    opened by davidfstein 3
  • Version update of Boruta on pypi?

    Version update of Boruta on pypi?

    Hello, I'd love to use new features like return_df in BorutaPy transform method, but can't with the latest version on pypi. Is there any chance that I can use soon? When is the next update?

    opened by proexcuse 0
  • PKG for the survival analysis

    PKG for the survival analysis

    Hi, I am trying to run the PKG for the survival analysis, keep having this issues, please help, thanks

    obj2 <- Boruta(Surv(data_wide2$ihd_time, data_wide2$ihd_ep),data_wide2 )

    Error in x[, i] <- frame[[i]] : number of items to replace is not a multiple of replacement length

    opened by MoMaz123 5
  • ImportError: cannot import name 'BorutaPy' from 'boruta'

    ImportError: cannot import name 'BorutaPy' from 'boruta'

    I download boruta's latest version through pypi (0.3 version) This error still occur (ImportError: cannot import name 'BorutaPy' from 'boruta' )

    Is there any solution for this error?

    My Ubuntu server specs are below Ubuntu 20.04.2 LTS python 3.7.11 conda 4.12.0

    opened by soemthlng 0
  • Implements sample_weight and optional permutation and SHAP importance, categorical features, boxplot

    Implements sample_weight and optional permutation and SHAP importance, categorical features, boxplot

    Hi,

    It took me a while but finally found the time to work on the continuation of the discussion https://github.com/scikit-learn-contrib/boruta_py/pull/77

    Meaning:

    • Not introducing new dependencies, a check import is performed if the User wants to use SHAP or get the matplotlib boxplot
    • Permutation importance (sklearn) is also implemented but optional and easy to switch off
    • Categorical features are encoded if any (optional)
    • sample_weight can now be passed to the fit method
    • A notebook illustrates the changes and compares the original Boruta_py and the new features
    • Add a note in the readme
    opened by ThomasBury 5
Owner
scikit-learn compatible projects
scikit-learn addon to operate on set/"group"-based features

skl-groups skl-groups is a package to perform machine learning on sets (or "groups") of features in Python. It extends the scikit-learn library with s

Danica J. Sutherland 41 Apr 06, 2022
open-source feature selection repository in python

scikit-feature Feature selection repository scikit-feature in Python. scikit-feature is an open-source feature selection repository in Python develope

Jundong Li 1.3k Jan 05, 2023
An open source python library for automated feature engineering

"One of the holy grails of machine learning is to automate more and more of the feature engineering process." ― Pedro Domingos, A Few Useful Things to

alteryx 6.4k Jan 05, 2023
A set of tools for creating and testing machine learning features, with a scikit-learn compatible API

Feature Forge This library provides a set of tools that can be useful in many machine learning applications (classification, clustering, regression, e

Machinalis 380 Nov 05, 2022
Python implementations of the Boruta all-relevant feature selection method.

boruta_py This project hosts Python implementations of the Boruta all-relevant feature selection method. Related blog post How to install Install with

1.2k Jan 04, 2023
A scikit-learn-compatible Python implementation of ReBATE, a suite of Relief-based feature selection algorithms for Machine Learning.

Master status: Development status: Package information: scikit-rebate This package includes a scikit-learn-compatible Python implementation of ReBATE,

Epistasis Lab at UPenn 374 Dec 15, 2022
A sklearn-compatible Python implementation of Multifactor Dimensionality Reduction (MDR) for feature construction.

Master status: Development status: Package information: MDR A scikit-learn-compatible Python implementation of Multifactor Dimensionality Reduction (M

Epistasis Lab at UPenn 122 Jul 06, 2022
A fast xgboost feature selection algorithm

BoostARoota A Fast XGBoost Feature Selection Algorithm (plus other sklearn tree-based classifiers) Why Create Another Algorithm? Automated processes l

Chase DeHan 187 Dec 22, 2022
a feature engineering wrapper for sklearn

Few Few is a Feature Engineering Wrapper for scikit-learn. Few looks for a set of feature transformations that work best with a specified machine lear

William La Cava 47 Nov 18, 2022
Automatic extraction of relevant features from time series:

tsfresh This repository contains the TSFRESH python package. The abbreviation stands for "Time Series Feature extraction based on scalable hypothesis

Blue Yonder GmbH 7k Jan 03, 2023