Use of mixed in Wrapper Methond¶
In [0]:
!pip install mlxtend
How it works¶
Sequential feature selection algorithms are a family of greedy search algorithms that are used to reduce an initial d-dimensional feature space to a k-dimensional feature subspace where k < d.
In as nutshell, SFAs remove or add one feature at the time based on the classifier performance until a feature subset of the disired size k is reached. There are 4 different flavors SFAs available via the SequentialFeatureSelector:
- Sequential Forward Slection (SFA)
- Sequential Backward Selection (SBS)
- Sequential Forward Floating Selection (SFFS)
- Sequential Backward Floating Selection (SBFS)
Step Foward Selection (SFS)¶
In [2]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
In [0]:
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.metrics import roc_auc_score
from mlxtend.feature_selection import SequentialFeatureSelector as SFS
In [0]:
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
In [0]:
data = load_wine()
In [6]:
data.keys()
Out[6]:
In [7]:
print(data.DESCR)
In [0]:
x = pd.DataFrame(data.data)
y = data.target
In [10]:
x.columns = data.feature_names
x.head()
Out[10]:
In [11]:
x.isnull().sum()
Out[11]:
In [12]:
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)
x_train.shape, x_test.shape
Out[12]:
Step Forward Feature Selection (SFS)¶
In [16]:
# 特徴量７つ抽出（７つ選んだ結果がプロパティに残るので、７つ選んだプロパティが最大とは限らない）
sfs = SFS(RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1),
k_features = 7,
forward = True,
floating = False,
verbose = 2,
scoring = 'accuracy',
cv = 4,
n_jobs = -1
).fit(x_train, y_train)
In [17]:
# 選択された7つの特徴量
sfs.k_feature_names_
Out[17]:
In [18]:
# 選択された７つの特徴量のカラム番号
sfs.k_feature_idx_
Out[18]:
In [19]:
# スコア
sfs.k_score_
Out[19]:
In [21]:
# 特徴量増加毎のスコアの変化確認
pd.DataFrame.from_dict(sfs.get_metric_dict()).T
Out[21]:
上記の結果から、最もスコアの高い特徴量の組を選んで予測モデルを作成すれば良い。
(上記では、特徴量数が３-magnesium, flavanoids, color_intensity で、スコア最良）
上記のやり方では、手動で最も高いスコアの組を読み取らなければならないが、以下のやり方だと、k_feature_names_プロパティに最も良い組み合わせが残る。
In [33]:
# 特徴量１つから８つの組み合わせの中でスコアが最大の組み合わせを見つける。
sfs = SFS(RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1),
k_features = (1,8),
forward = True,
floating = False,
verbose = 2,
scoring = 'accuracy',
cv = 4,
n_jobs = -1
).fit(x_train, y_train)
In [23]:
sfs.k_score_
Out[23]:
In [24]:
sfs.k_feature_names_
Out[24]:
上のように、最もスコアの高い組み合わせが抽出された。
Step Backward Selection (SBS)¶
In [39]:
# 特徴量８つから１つまでの組み合わせの中でスコアが最大の組み合わせを見つける。
sbs = SFS(RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1),
k_features = (1, 8),
forward = False,
floating = False,
verbose = 2,
scoring = 'accuracy',
cv = 4,
n_jobs = -1
).fit(x_train.values, y_train)
# ※：StepBackward(forward = False)に変えてpd.DataFrameを渡すとエラーになるので、x_train.valuesで渡す
# エラー詳細url： https://github.com/rasbt/mlxtend/issues/505
In [40]:
sbs.k_score_
Out[40]:
In [41]:
sbs.k_feature_names_
# DataFrameのvaluesを渡しているので、インデックスが出る
Out[41]:
Exhausitive Feature Selection (EFS)¶
In [0]:
from mlxtend.feature_selection import ExhaustiveFeatureSelector as EFS
In [47]:
# 最小特徴量４、最大特徴量５で探す
efs = EFS(RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1),
min_features = 4,
max_features = 5,
scoring = 'accuracy',
cv= None,
n_jobs = -1
).fit(x_train, y_train)
上記の場合、１３個から４つ選ぶ組み合わせと、１３個から５つ選ぶ組み合わせを探すので、全部で２００２組の組み合わせから探すことになる。なので、計算コストが膨大で時間がかかる。
In [46]:
# １３個から４個選ぶ組み合わせと１３個から５つ選ぶ組み合わせの数
from scipy.special import comb
comb(13, 4, exact=True) + comb(13, 5, exact=True)
Out[46]:
In [48]:
efs.best_score_
Out[48]:
In [49]:
efs.best_feature_names_
Out[49]:
In [50]:
efs.best_idx_
Out[50]:
In [0]:
from mlxtend.plotting import plot_sequential_feature_selection as plot_sfs
In [52]:
plot_sfs(efs.get_metric_dict(), kind='std_dev')
Out[52]:
In [0]: