我试图使用scikit-learn的LabelEncoder来编码字符串标签的pandas DataFrame。由于数据帧有许多(50+)列,我想避免为每一列创建一个LabelEncoder对象;我宁愿只有一个大的LabelEncoder对象,它可以跨所有数据列工作。

将整个DataFrame扔到LabelEncoder中会产生以下错误。请记住,我在这里使用的是虚拟数据;实际上,我正在处理大约50列的字符串标记数据,所以需要一个解决方案,不引用任何列的名称。

import pandas
from sklearn import preprocessing 

df = pandas.DataFrame({
    'pets': ['cat', 'dog', 'cat', 'monkey', 'dog', 'dog'], 
    'owner': ['Champ', 'Ron', 'Brick', 'Champ', 'Veronica', 'Ron'], 
    'location': ['San_Diego', 'New_York', 'New_York', 'San_Diego', 'San_Diego', 
                 'New_York']
})

le = preprocessing.LabelEncoder()

le.fit(df)

回溯(最近一次调用): 文件“”,第1行,在 文件"/Users/bbalin/anaconda/lib/python2.7/site-packages/sklearn/预处理/label.py",第103行 y = column_or_1d(y, warn=True) 文件"/Users/bbalin/anaconda/lib/python2.7/site-packages/sklearn/utils/validation.py",第306行,在column_or_1d中 raise ValueError("错误的输入形状{0}".format(形状)) ValueError:错误的输入形状(6,3)

对于如何解决这个问题有什么想法吗?


当前回答

这是我解决你问题的办法。为了将包含文本的数据帧列转换为编码值,只需使用我的函数text_to_numbers,它返回LE的字典。Key是列LabelEncoder()作为值的列名。

def text_to_numbers(df):
        le_dict = dict()
        for i in df.columns:
            if df[i].dtype not in ["float64", "bool", "int64"]:
                le_dict[i] = preprocessing.LabelEncoder()
                df[i] = le_dict[i].fit_transform(df[i])
    
        return df, le_dict

下面的函数将使保留原始的未编码数据帧成为可能。

 def numbers_to_text(df, le_dict):
        for i in le_dict.keys():
            df[i] = le_dict[i].inverse_transform(df[i])
    
        return df

其他回答

我们不需要LabelEncoder。

您可以将列转换为类别,然后获取它们的代码。我使用下面的字典推导将此过程应用于每一列,并将结果包装回具有相同索引和列名的相同形状的数据框架中。

>>> pd.DataFrame({col: df[col].astype('category').cat.codes for col in df}, index=df.index)
   location  owner  pets
0         1      1     0
1         0      2     1
2         0      0     0
3         1      1     2
4         1      3     1
5         0      2     1

要创建映射字典,你可以使用字典理解式枚举类别:

>>> {col: {n: cat for n, cat in enumerate(df[col].astype('category').cat.categories)} 
     for col in df}

{'location': {0: 'New_York', 1: 'San_Diego'},
 'owner': {0: 'Brick', 1: 'Champ', 2: 'Ron', 3: 'Veronica'},
 'pets': {0: 'cat', 1: 'dog', 2: 'monkey'}}

这是一年半后的事实,但我也需要能够。transform()多个熊猫数据帧列一次(以及能够。inverse_transform()他们)。这扩展了上面@PriceHardman的优秀建议:

class MultiColumnLabelEncoder(LabelEncoder):
    """
    Wraps sklearn LabelEncoder functionality for use on multiple columns of a
    pandas dataframe.

    """
    def __init__(self, columns=None):
        self.columns = columns

    def fit(self, dframe):
        """
        Fit label encoder to pandas columns.

        Access individual column classes via indexig `self.all_classes_`

        Access individual column encoders via indexing
        `self.all_encoders_`
        """
        # if columns are provided, iterate through and get `classes_`
        if self.columns is not None:
            # ndarray to hold LabelEncoder().classes_ for each
            # column; should match the shape of specified `columns`
            self.all_classes_ = np.ndarray(shape=self.columns.shape,
                                           dtype=object)
            self.all_encoders_ = np.ndarray(shape=self.columns.shape,
                                            dtype=object)
            for idx, column in enumerate(self.columns):
                # fit LabelEncoder to get `classes_` for the column
                le = LabelEncoder()
                le.fit(dframe.loc[:, column].values)
                # append the `classes_` to our ndarray container
                self.all_classes_[idx] = (column,
                                          np.array(le.classes_.tolist(),
                                                  dtype=object))
                # append this column's encoder
                self.all_encoders_[idx] = le
        else:
            # no columns specified; assume all are to be encoded
            self.columns = dframe.iloc[:, :].columns
            self.all_classes_ = np.ndarray(shape=self.columns.shape,
                                           dtype=object)
            for idx, column in enumerate(self.columns):
                le = LabelEncoder()
                le.fit(dframe.loc[:, column].values)
                self.all_classes_[idx] = (column,
                                          np.array(le.classes_.tolist(),
                                                  dtype=object))
                self.all_encoders_[idx] = le
        return self

    def fit_transform(self, dframe):
        """
        Fit label encoder and return encoded labels.

        Access individual column classes via indexing
        `self.all_classes_`

        Access individual column encoders via indexing
        `self.all_encoders_`

        Access individual column encoded labels via indexing
        `self.all_labels_`
        """
        # if columns are provided, iterate through and get `classes_`
        if self.columns is not None:
            # ndarray to hold LabelEncoder().classes_ for each
            # column; should match the shape of specified `columns`
            self.all_classes_ = np.ndarray(shape=self.columns.shape,
                                           dtype=object)
            self.all_encoders_ = np.ndarray(shape=self.columns.shape,
                                            dtype=object)
            self.all_labels_ = np.ndarray(shape=self.columns.shape,
                                          dtype=object)
            for idx, column in enumerate(self.columns):
                # instantiate LabelEncoder
                le = LabelEncoder()
                # fit and transform labels in the column
                dframe.loc[:, column] =\
                    le.fit_transform(dframe.loc[:, column].values)
                # append the `classes_` to our ndarray container
                self.all_classes_[idx] = (column,
                                          np.array(le.classes_.tolist(),
                                                  dtype=object))
                self.all_encoders_[idx] = le
                self.all_labels_[idx] = le
        else:
            # no columns specified; assume all are to be encoded
            self.columns = dframe.iloc[:, :].columns
            self.all_classes_ = np.ndarray(shape=self.columns.shape,
                                           dtype=object)
            for idx, column in enumerate(self.columns):
                le = LabelEncoder()
                dframe.loc[:, column] = le.fit_transform(
                        dframe.loc[:, column].values)
                self.all_classes_[idx] = (column,
                                          np.array(le.classes_.tolist(),
                                                  dtype=object))
                self.all_encoders_[idx] = le
        return dframe.loc[:, self.columns].values

    def transform(self, dframe):
        """
        Transform labels to normalized encoding.
        """
        if self.columns is not None:
            for idx, column in enumerate(self.columns):
                dframe.loc[:, column] = self.all_encoders_[
                    idx].transform(dframe.loc[:, column].values)
        else:
            self.columns = dframe.iloc[:, :].columns
            for idx, column in enumerate(self.columns):
                dframe.loc[:, column] = self.all_encoders_[idx]\
                    .transform(dframe.loc[:, column].values)
        return dframe.loc[:, self.columns].values

    def inverse_transform(self, dframe):
        """
        Transform labels back to original encoding.
        """
        if self.columns is not None:
            for idx, column in enumerate(self.columns):
                dframe.loc[:, column] = self.all_encoders_[idx]\
                    .inverse_transform(dframe.loc[:, column].values)
        else:
            self.columns = dframe.iloc[:, :].columns
            for idx, column in enumerate(self.columns):
                dframe.loc[:, column] = self.all_encoders_[idx]\
                    .inverse_transform(dframe.loc[:, column].values)
        return dframe.loc[:, self.columns].values

例子:

如果df和df_copy()是混合类型的pandas数据帧,你可以将MultiColumnLabelEncoder()应用到dtype=object列上,方法如下:

# get `object` columns
df_object_columns = df.iloc[:, :].select_dtypes(include=['object']).columns
df_copy_object_columns = df_copy.iloc[:, :].select_dtypes(include=['object']).columns

# instantiate `MultiColumnLabelEncoder`
mcle = MultiColumnLabelEncoder(columns=object_columns)

# fit to `df` data
mcle.fit(df)

# transform the `df` data
mcle.transform(df)

# returns output like below
array([[1, 0, 0, ..., 1, 1, 0],
       [0, 5, 1, ..., 1, 1, 2],
       [1, 1, 1, ..., 1, 1, 2],
       ..., 
       [3, 5, 1, ..., 1, 1, 2],

# transform `df_copy` data
mcle.transform(df_copy)

# returns output like below (assuming the respective columns 
# of `df_copy` contain the same unique values as that particular 
# column in `df`
array([[1, 0, 0, ..., 1, 1, 0],
       [0, 5, 1, ..., 1, 1, 2],
       [1, 1, 1, ..., 1, 1, 2],
       ..., 
       [3, 5, 1, ..., 1, 1, 2],

# inverse `df` data
mcle.inverse_transform(df)

# outputs data like below
array([['August', 'Friday', '2013', ..., 'N', 'N', 'CA'],
       ['April', 'Tuesday', '2014', ..., 'N', 'N', 'NJ'],
       ['August', 'Monday', '2014', ..., 'N', 'N', 'NJ'],
       ..., 
       ['February', 'Tuesday', '2014', ..., 'N', 'N', 'NJ'],
       ['April', 'Tuesday', '2014', ..., 'N', 'N', 'NJ'],
       ['March', 'Tuesday', '2013', ..., 'N', 'N', 'NJ']], dtype=object)

# inverse `df_copy` data
mcle.inverse_transform(df_copy)

# outputs data like below
array([['August', 'Friday', '2013', ..., 'N', 'N', 'CA'],
       ['April', 'Tuesday', '2014', ..., 'N', 'N', 'NJ'],
       ['August', 'Monday', '2014', ..., 'N', 'N', 'NJ'],
       ..., 
       ['February', 'Tuesday', '2014', ..., 'N', 'N', 'NJ'],
       ['April', 'Tuesday', '2014', ..., 'N', 'N', 'NJ'],
       ['March', 'Tuesday', '2013', ..., 'N', 'N', 'NJ']], dtype=object)

你可以通过索引访问单独的列类、列标签和用于适合每个列的列编码器:

mcle.all_classes_ mcle.all_encoders_ mcle.all_labels_

假设你只是想获得一个sklearn.预处理. labelencoder()对象,可以用来表示你的列,你所要做的就是:

le.fit(df.columns)

在上面的代码中,每一列都有一个唯一的数字。 更精确地说,你将得到df的1:1映射。列到le.transform(df.columns.get_values())。要获得列的编码,只需将其传递给le.transform(…)。作为一个例子,下面将得到每一列的编码:

le.transform(df.columns.get_values())

假设你想为你所有的行标签创建一个sklearn.预处理. labelencoder()对象,你可以这样做:

le.fit([y for x in df.get_values() for y in x])

在本例中,您很可能拥有非唯一的行标签(如您的问题所示)。要查看编码器创建了哪些类,可以执行le.classes_。你会注意到,这应该具有与set中相同的元素(y for x in df.get_values() for y in x)。再次使用le.transform(…)将行标签转换为编码标签。例如,如果您想检索df. xml文件中第一列的标签。列数组和第一行,你可以这样做:

le.transform([df.get_value(0, df.columns[0])])

你在评论中提出的问题有点复杂,但仍然可以 完成:

le.fit([str(z) for z in set((x[0], y) for x in df.iteritems() for y in x[1])])

上面的代码实现了以下功能:

使所有(列,行)对的唯一组合 将每个对表示为元组的字符串版本。这是克服LabelEncoder类不支持元组作为类名的一种变通方法。 将新项目贴合到LabelEncoder。

现在要使用这个新模型就有点复杂了。假设我们想要提取在前一个例子中查找的同一项的表示(df中的第一列)。列和第一行),我们可以这样做:

le.transform([str((df.columns[0], df.get_value(0, df.columns[0])))])

记住,现在每个查找都是一个元组的字符串表示 包含(列、行)。

不,LabelEncoder不这样做。它接受类标签的1维数组并生成1维数组。它的设计目的是处理分类问题中的类标签,而不是任意数据,任何强迫它用于其他用途的尝试都需要代码将实际问题转换为它解决的问题(并将解决方案转换回原始空间)。

如果我们有单列来做标签编码和它的逆变换,当python中有多列时,很容易做到这一点

def stringtocategory(dataset):
    '''
    @author puja.sharma
    @see The function label encodes the object type columns and gives label      encoded and inverse tranform of the label encoded data
    @param dataset dataframe on whoes column the label encoding has to be done
    @return label encoded and inverse tranform of the label encoded data.
   ''' 
   data_original = dataset[:]
   data_tranformed = dataset[:]
   for y in dataset.columns:
       #check the dtype of the column object type contains strings or chars
       if (dataset[y].dtype == object):
          print("The string type features are  : " + y)
          le = preprocessing.LabelEncoder()
          le.fit(dataset[y].unique())
          #label encoded data
          data_tranformed[y] = le.transform(dataset[y])
          #inverse label transform  data
          data_original[y] = le.inverse_transform(data_tranformed[y])
   return data_tranformed,data_original