我有一个熊猫数据框架，我想把它分为3个单独的集。我知道使用sklearn中的train_test_split。交叉验证，可以将数据分为两组(训练和测试)。然而，我无法找到将数据分成三组的任何解决方案。最好是有原始数据的下标。
我知道一个解决办法是使用train_test_split两次，并以某种方式调整索引。但是是否有一种更标准/内置的方法将数据分成3组而不是2组?
当前回答
我能想到的最简单的方法是将分割分数映射到数组下标，如下所示:
train_set = data[:int((len(data)+1)*train_fraction)]
test_set = data[int((len(data)+1)*train_fraction):int((len(data)+1)*(train_fraction+test_fraction))]
val_set = data[int((len(data)+1)*(train_fraction+test_fraction)):]
其中data = random.shuffle(data)
其他回答
将数据集分割为训练集和测试集，如在其他答案中一样，使用
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
然后，如果您适合您的模型，您可以添加validation_split作为参数。这样就不需要提前创建验证集。例如:
from tensorflow.keras import Model
model = Model(input_layer, out)
[...]
history = model.fit(x=X_train, y=y_train, [...], validation_split = 0.3)
验证集旨在作为训练集训练期间的代表运行测试集，完全来自训练集，无论是通过k-fold交叉验证(推荐)还是通过validation_split;然后，您不需要单独创建一个验证集，仍然可以将数据集分为您所要求的三个集。
下面是一个Python函数，它将Pandas数据帧划分为分层抽样的训练、验证和测试数据帧。它通过两次调用scikit-learn的函数train_test_split()来执行这种分割。
import pandas as pd
from sklearn.model_selection import train_test_split
def split_stratified_into_train_val_test(df_input, stratify_colname='y',
frac_train=0.6, frac_val=0.15, frac_test=0.25,
random_state=None):
'''
Splits a Pandas dataframe into three subsets (train, val, and test)
following fractional ratios provided by the user, where each subset is
stratified by the values in a specific column (that is, each subset has
the same relative frequency of the values in the column). It performs this
splitting by running train_test_split() twice.
Parameters
----------
df_input : Pandas dataframe
Input dataframe to be split.
stratify_colname : str
The name of the column that will be used for stratification. Usually
this column would be for the label.
frac_train : float
frac_val : float
frac_test : float
The ratios with which the dataframe will be split into train, val, and
test data. The values should be expressed as float fractions and should
sum to 1.0.
random_state : int, None, or RandomStateInstance
Value to be passed to train_test_split().
Returns
-------
df_train, df_val, df_test :
Dataframes containing the three splits.
'''
if frac_train + frac_val + frac_test != 1.0:
raise ValueError('fractions %f, %f, %f do not add up to 1.0' % \
(frac_train, frac_val, frac_test))
if stratify_colname not in df_input.columns:
raise ValueError('%s is not a column in the dataframe' % (stratify_colname))
X = df_input # Contains all columns.
y = df_input[[stratify_colname]] # Dataframe of just the column on which to stratify.
# Split original dataframe into train and temp dataframes.
df_train, df_temp, y_train, y_temp = train_test_split(X,
y,
stratify=y,
test_size=(1.0 - frac_train),
random_state=random_state)
# Split the temp dataframe into val and test dataframes.
relative_frac_test = frac_test / (frac_val + frac_test)
df_val, df_test, y_val, y_test = train_test_split(df_temp,
y_temp,
stratify=y_temp,
test_size=relative_frac_test,
random_state=random_state)
assert len(df_input) == len(df_train) + len(df_val) + len(df_test)
return df_train, df_val, df_test
下面是一个完整的工作示例。
考虑一个数据集，该数据集具有一个标签，您希望在其上执行分层。这个标签在原始数据集中有自己的分布，比如75% foo, 15% bar和10% baz。现在，让我们使用60/20/20的比例将数据集分割为训练、验证和测试子集，其中每个分割保留相同的标签分布。如下图所示:
下面是示例数据集:
df = pd.DataFrame( { 'A': list(range(0, 100)),
'B': list(range(100, 0, -1)),
'label': ['foo'] * 75 + ['bar'] * 15 + ['baz'] * 10 } )
df.head()
# A B label
# 0 0 100 foo
# 1 1 99 foo
# 2 2 98 foo
# 3 3 97 foo
# 4 4 96 foo
df.shape
# (100, 3)
df.label.value_counts()
# foo 75
# bar 15
# baz 10
# Name: label, dtype: int64
现在，让我们从上面调用split_stratified_into_train_val_test()函数，按照60/20/20的比例获取训练、验证和测试数据帧。
df_train, df_val, df_test = \
split_stratified_into_train_val_test(df, stratify_colname='label', frac_train=0.60, frac_val=0.20, frac_test=0.20)
三个数据帧df_train、df_val和df_test包含所有原始行，但它们的大小将遵循上面的比例。
df_train.shape
#(60, 3)
df_val.shape
#(20, 3)
df_test.shape
#(20, 3)
此外，三次分割中的每一次都将具有相同的标签分布，即75% foo, 15% bar和10% baz。
df_train.label.value_counts()
# foo 45
# bar 9
# baz 6
# Name: label, dtype: int64
df_val.label.value_counts()
# foo 15
# bar 3
# baz 2
# Name: label, dtype: int64
df_test.label.value_counts()
# foo 15
# bar 3
# baz 2
# Name: label, dtype: int64
注意:
函数被编写来处理随机集创建的播种。你不应该依赖集分割，它不会随机化集合。
import numpy as np
import pandas as pd
def train_validate_test_split(df, train_percent=.6, validate_percent=.2, seed=None):
np.random.seed(seed)
perm = np.random.permutation(df.index)
m = len(df.index)
train_end = int(train_percent * m)
validate_end = int(validate_percent * m) + train_end
train = df.iloc[perm[:train_end]]
validate = df.iloc[perm[train_end:validate_end]]
test = df.iloc[perm[validate_end:]]
return train, validate, test
示范
np.random.seed([3,1415])
df = pd.DataFrame(np.random.rand(10, 5), columns=list('ABCDE'))
df
train, validate, test = train_validate_test_split(df)
train
validate
test
我能想到的最简单的方法是将分割分数映射到数组下标，如下所示:
train_set = data[:int((len(data)+1)*train_fraction)]
test_set = data[int((len(data)+1)*train_fraction):int((len(data)+1)*(train_fraction+test_fraction))]
val_set = data[int((len(data)+1)*(train_fraction+test_fraction)):]
其中data = random.shuffle(data)
然而，将数据集分为train、test、cv(0.6、0.2、0.2)的一种方法是使用train_test_split方法两次。
from sklearn.model_selection import train_test_split
x, x_test, y, y_test = train_test_split(xtrain,labels,test_size=0.2,train_size=0.8)
x_train, x_cv, y_train, y_cv = train_test_split(x,y,test_size = 0.25,train_size =0.75)
