@kalu的回答更新了一下，用了最新的熊猫:

def find_duplicated_columns(df):
    dupes = []

    columns = df.columns

    for i in range(len(columns)):
        col1 = df.iloc[:, i]
        for j in range(i + 1, len(columns)):
            col2 = df.iloc[:, j]
            # break early if dtypes aren't the same (helps deal with
            # categorical dtypes)
            if col1.dtype is not col2.dtype:
                break
            # otherwise compare values
            if col1.equals(col2):
                dupes.append(columns[i])
                break

    return dupes

下面的方法将识别dupe列，以检查最初构建数据框架时出错的地方。

dupes = pd.DataFrame(df.columns)
dupes[dupes.duplicated()]

转置对于大数据帧来说效率很低。这里有一个替代方案:

def duplicate_columns(frame):
    groups = frame.columns.to_series().groupby(frame.dtypes).groups
    dups = []
    for t, v in groups.items():
        dcols = frame[v].to_dict(orient="list")

        vs = dcols.values()
        ks = dcols.keys()
        lvs = len(vs)

        for i in range(lvs):
            for j in range(i+1,lvs):
                if vs[i] == vs[j]: 
                    dups.append(ks[i])
                    break

    return dups       

像这样使用它:

dups = duplicate_columns(frame)
frame = frame.drop(dups, axis=1)

Edit

一个内存高效的版本，像对待其他值一样对待nan:

from pandas.core.common import array_equivalent

def duplicate_columns(frame):
    groups = frame.columns.to_series().groupby(frame.dtypes).groups
    dups = []

    for t, v in groups.items():

        cs = frame[v].columns
        vs = frame[v]
        lcs = len(cs)

        for i in range(lcs):
            ia = vs.iloc[:,i].values
            for j in range(i+1, lcs):
                ja = vs.iloc[:,j].values
                if array_equivalent(ia, ja):
                    dups.append(cs[i])
                    break

    return dups

@kalu的回答更新了一下，用了最新的熊猫:

def find_duplicated_columns(df):
    dupes = []

    columns = df.columns

    for i in range(len(columns)):
        col1 = df.iloc[:, i]
        for j in range(i + 1, len(columns)):
            col2 = df.iloc[:, j]
            # break early if dtypes aren't the same (helps deal with
            # categorical dtypes)
            if col1.dtype is not col2.dtype:
                break
            # otherwise compare values
            if col1.equals(col2):
                dupes.append(columns[i])
                break

    return dupes

以防有人还在寻找如何在Python中为Pandas数据帧的列中寻找重复值的答案，我想出了这个解决方案:

def get_dup_columns(m):
    '''
    This will check every column in data frame 
    and verify if you have duplicated columns.
    can help whenever you are cleaning big data sets of 50+ columns 
    and clean up a little  bit for you
    The result will be a list of tuples showing what columns are duplicates
    for example
    (column A, Column C)
    That means that column A is duplicated with column C
    more info go to https://wanatux.com
    '''
    headers_list = [x for x in m.columns]
    duplicate_col2 = []
    y = 0
    while y <= len(headers_list)-1:
        for x in range(1,len(headers_list)-1):
            if m[headers_list[y]].equals(m[headers_list[x]]) == False:        
                continue
            else:
                duplicate_col2.append((headers_list[y],headers_list[x]))
        headers_list.pop(0)  
    return duplicate_col2

你可以像这样强制转换定义:

duplicate_col = get_dup_columns(pd_excel)

它将显示如下结果:

 [('column a', 'column k'),
 ('column a', 'column r'),
 ('column h', 'column m'),
 ('column k', 'column r')]

快速和简单的方法删除复制列的值:

df = df.T.drop_duplicates().T

更多信息:Pandas DataFrame drop_duplicate manual。