我如何才能实现SQL的IN和NOT IN的等价?

我有一个所需值的列表。 场景如下:

df = pd.DataFrame({'country': ['US', 'UK', 'Germany', 'China']})
countries_to_keep = ['UK', 'China']

# pseudo-code:
df[df['country'] not in countries_to_keep]

我目前的做法如下:

df = pd.DataFrame({'country': ['US', 'UK', 'Germany', 'China']})
df2 = pd.DataFrame({'country': ['UK', 'China'], 'matched': True})

# IN
df.merge(df2, how='inner', on='country')

# NOT IN
not_in = df.merge(df2, how='left', on='country')
not_in = not_in[pd.isnull(not_in['matched'])]

但这似乎是一个可怕的拼凑。有人能改进吗?


当前回答

我的2c价值: 我需要一个数据框架的in和ifelse语句的组合,这对我来说很有用。

sale_method = pd.DataFrame(model_data["Sale Method"].str.upper())
sale_method["sale_classification"] = np.where(
    sale_method["Sale Method"].isin(["PRIVATE"]),
    "private",
    np.where(
        sale_method["Sale Method"].str.contains("AUCTION"), "auction", "other"
    ),
)

其他回答

df = pd.DataFrame({'countries':['US','UK','Germany','China']})
countries = ['UK','China']

实现:

df[df.countries.isin(countries)]

实施不像在其他国家:

df[df.countries.isin([x for x in np.unique(df.countries) if x not in countries])]

使用.query()方法的替代解决方案:

In [5]: df.query("countries in @countries_to_keep")
Out[5]:
  countries
1        UK
3     China

In [6]: df.query("countries not in @countries_to_keep")
Out[6]:
  countries
0        US
2   Germany

你也可以在.query()中使用.isin():

df.query('country.isin(@countries_to_keep).values')

# Or alternatively:
df.query('country.isin(["UK", "China"]).values')

要否定你的查询,使用~:

df.query('~country.isin(@countries_to_keep).values')

更新:

另一种方法是使用比较操作符:

df.query('country == @countries_to_keep')

# Or alternatively:
df.query('country == ["UK", "China"]')

要对查询求反,使用!=:

df.query('country != @countries_to_keep')

我的2c价值: 我需要一个数据框架的in和ifelse语句的组合,这对我来说很有用。

sale_method = pd.DataFrame(model_data["Sale Method"].str.upper())
sale_method["sale_classification"] = np.where(
    sale_method["Sale Method"].isin(["PRIVATE"]),
    "private",
    np.where(
        sale_method["Sale Method"].str.contains("AUCTION"), "auction", "other"
    ),
)

为什么没有人谈论各种过滤方法的性能?事实上,这个主题经常出现在这里(参见示例)。我为一个大型数据集做了自己的性能测试。这是非常有趣和有教育意义的。

df = pd.DataFrame({'animals': np.random.choice(['cat', 'dog', 'mouse', 'birds'], size=10**7), 
                   'number': np.random.randint(0,100, size=(10**7,))})

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 2 columns):
 #   Column   Dtype 
---  ------   ----- 
 0   animals  object
 1   number   int64 
dtypes: int64(1), object(1)
memory usage: 152.6+ MB
%%timeit
# .isin() by one column
conditions = ['cat', 'dog']
df[df.animals.isin(conditions)]
367 ms ± 2.34 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
# .query() by one column
conditions = ['cat', 'dog']
df.query('animals in @conditions')
395 ms ± 3.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
# .loc[]
df.loc[(df.animals=='cat')|(df.animals=='dog')]
987 ms ± 5.17 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
df[df.apply(lambda x: x['animals'] in ['cat', 'dog'], axis=1)]
41.9 s ± 490 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
new_df = df.set_index('animals')
new_df.loc[['cat', 'dog'], :]
3.64 s ± 62.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
new_df = df.set_index('animals')
new_df[new_df.index.isin(['cat', 'dog'])]
469 ms ± 8.98 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
s = pd.Series(['cat', 'dog'], name='animals')
df.merge(s, on='animals', how='inner')
796 ms ± 30.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

因此,isin方法是最快的,而带有apply()的方法是最慢的,这并不奇怪。