如果你想用try和except语句检查两列，这个可以帮你。

if "column_2" in df.columns:
    try:
        df[['column_1', "column_2"]] = df[['header', "column_2"]].drop_duplicates(subset = ["column_2", "column_1"] ,keep="first")
    except:
        df[["column_2"]] = df[["column_2"]].drop_duplicates(subset="column_2" ,keep="first")
        print(f"No column_1 for {path}.")
try:
    df[["column_1"]] = df[["column_1"]].drop_duplicates(subset="column_1" ,keep="first")
except:
    print(f"No column_1 or column_2 for {path}.")

如果你想将结果存储在另一个数据集中:

df.drop_duplicates(keep=False)

or

df.drop_duplicates(keep=False, inplace=False)

如果需要更新相同的数据集:

df.drop_duplicates(keep=False, inplace=True)

上面的例子将删除所有重复项并保留一个，类似于SQL中的DISTINCT *

只是想添加到Ben关于drop_duplicate的答案:

keep: {' first '， ' last '， False}，默认' first '

first:删除除第一次出现之外的重复项。 last:删除除最后一次出现之外的重复项。 False:删除所有副本。

所以将keep设置为False会给你想要的答案。

DataFrame.drop_duplicates(*args, **kwargs) Return DataFrame with duplicate rows removed, optionally only considering certain columns Parameters: subset : column label or sequence of labels, optional Only consider certain columns for identifying duplicates, by default use all of the columns keep : {‘first’, ‘last’, False}, default ‘first’ first : Drop duplicates except for the first occurrence. last : Drop duplicates except for the last occurrence. False : Drop all duplicates. take_last : deprecated inplace : boolean, default False Whether to drop duplicates in place or to return a copy cols : kwargs only argument of subset [deprecated] Returns: deduplicated : DataFrame

使用groupby和filter

import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.groupby(["A", "C"]).filter(lambda df:df.shape[0] == 1)

实际上，删除行0和1只需要(任何包含匹配的A和C的观测值都被保留):

In [335]:

df['AC']=df.A+df.C
In [336]:

print df.drop_duplicates('C', take_last=True) #this dataset is a special case, in general, one may need to first drop_duplicates by 'c' and then by 'a'.
     A  B  C    AC
2  foo  1  B  fooB
3  bar  1  A  barA

[2 rows x 4 columns]

但我怀疑你真正想要的是这个(一个包含匹配的A和C的观察结果被保留):

In [337]:

print df.drop_duplicates('AC')
     A  B  C    AC
0  foo  0  A  fooA
2  foo  1  B  fooB
3  bar  1  A  barA

[3 rows x 4 columns]

编辑:

因此，现在情况清楚多了:

In [352]:
DG=df.groupby(['A', 'C'])   
print pd.concat([DG.get_group(item) for item, value in DG.groups.items() if len(value)==1])
     A  B  C
2  foo  1  B
3  bar  1  A

[2 rows x 3 columns]

您可以使用replicated()标记所有重复的行，并过滤掉标记的行。如果你以后需要将列赋值给new_df，确保调用.copy()，这样你以后就不会得到SettingWithCopyWarning。

new_df = df[~df.duplicated(subset=['A', 'C'], keep=False)].copy()

该方法的一个很好的特性是，您可以有条件地使用它删除重复项。例如，仅当列A等于'foo'时删除所有重复的行，您可以使用以下代码。

new_df = df[~( df.duplicated(subset=['A', 'B', 'C'], keep=False) & df['A'].eq('foo') )].copy()

此外，如果您不希望按名称写出列，您可以传递df的切片。列到子集=。对于drop_duplicate()也是如此。

# to consider all columns for identifying duplicates
df[~df.duplicated(subset=df.columns, keep=False)].copy()

# the same is true for drop_duplicates
df.drop_duplicates(subset=df.columns, keep=False)

# to consider columns in positions 0 and 2 (i.e. 'A' and 'C') for identifying duplicates
df.drop_duplicates(subset=df.columns[[0, 2]], keep=False)