pandas drop_duplicate函数对于“唯一化”一个数据帧非常有用。我想删除在列的子集上重复的所有行。这可能吗?

    A   B   C
0   foo 0   A
1   foo 1   A
2   foo 1   B
3   bar 1   A

例如,我想删除与列A和C匹配的行,因此这应该删除行0和1。


当前回答

如果你想用try和except语句检查两列,这个可以帮你。

if "column_2" in df.columns:
    try:
        df[['column_1', "column_2"]] = df[['header', "column_2"]].drop_duplicates(subset = ["column_2", "column_1"] ,keep="first")
    except:
        df[["column_2"]] = df[["column_2"]].drop_duplicates(subset="column_2" ,keep="first")
        print(f"No column_1 for {path}.")
try:
    df[["column_1"]] = df[["column_1"]].drop_duplicates(subset="column_1" ,keep="first")
except:
    print(f"No column_1 or column_2 for {path}.")

其他回答

试试这些不同的方法

df = pd.DataFrame({"A":["foo", "foo", "foo", "bar","foo"], "B":[0,1,1,1,1], "C":["A","A","B","A","A"]})

>>>df.drop_duplicates( "A" , keep='first')

or

>>>df.drop_duplicates( keep='first')

or

>>>df.drop_duplicates( keep='last')

使用drop_duplicate和keep参数,这在pandas中要容易得多。

import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.drop_duplicates(subset=['A', 'C'], keep=False)

实际上,删除行0和1只需要(任何包含匹配的A和C的观测值都被保留):

In [335]:

df['AC']=df.A+df.C
In [336]:

print df.drop_duplicates('C', take_last=True) #this dataset is a special case, in general, one may need to first drop_duplicates by 'c' and then by 'a'.
     A  B  C    AC
2  foo  1  B  fooB
3  bar  1  A  barA

[2 rows x 4 columns]

但我怀疑你真正想要的是这个(一个包含匹配的A和C的观察结果被保留):

In [337]:

print df.drop_duplicates('AC')
     A  B  C    AC
0  foo  0  A  fooA
2  foo  1  B  fooB
3  bar  1  A  barA

[3 rows x 4 columns]

编辑:

因此,现在情况清楚多了:

In [352]:
DG=df.groupby(['A', 'C'])   
print pd.concat([DG.get_group(item) for item, value in DG.groups.items() if len(value)==1])
     A  B  C
2  foo  1  B
3  bar  1  A

[2 rows x 3 columns]

如果你想用try和except语句检查两列,这个可以帮你。

if "column_2" in df.columns:
    try:
        df[['column_1', "column_2"]] = df[['header', "column_2"]].drop_duplicates(subset = ["column_2", "column_1"] ,keep="first")
    except:
        df[["column_2"]] = df[["column_2"]].drop_duplicates(subset="column_2" ,keep="first")
        print(f"No column_1 for {path}.")
try:
    df[["column_1"]] = df[["column_1"]].drop_duplicates(subset="column_1" ,keep="first")
except:
    print(f"No column_1 or column_2 for {path}.")

使用groupby和filter

import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.groupby(["A", "C"]).filter(lambda df:df.shape[0] == 1)