假设我有一个df，它的列是" ID " " col_1 " " col_2 "我定义了一个函数:
F = x, y: my_function_expression。
现在我想应用f到df的两个列'col_1'， 'col_2'来逐个元素计算一个新列'col_3'，有点像:
df['col_3'] = df[['col_1','col_2']].apply(f)
# Pandas gives : TypeError: ('<lambda>() takes exactly 2 arguments (1 given)'
怎么办?
**添加详细示例如下***
import pandas as pd
df = pd.DataFrame({'ID':['1','2','3'], 'col_1': [0,2,3], 'col_2':[1,4,5]})
mylist = ['a','b','c','d','e','f']
def get_sublist(sta,end):
return mylist[sta:end+1]
#df['col_3'] = df[['col_1','col_2']].apply(get_sublist,axis=1)
# expect above to output df as below
ID col_1 col_2 col_3
0 1 0 1 ['a', 'b']
1 2 2 4 ['c', 'd', 'e']
2 3 3 5 ['d', 'e', 'f']
在Pandas中有一个简单的方法:
df['col_3'] = df.apply(lambda x: f(x.col_1, x.col_2), axis=1)
这允许f是一个用户定义的具有多个输入值的函数，并使用(安全的)列名而不是(不安全的)数字索引来访问列。
数据示例(基于原始问题):
import pandas as pd
df = pd.DataFrame({'ID':['1', '2', '3'], 'col_1': [0, 2, 3], 'col_2':[1, 4, 5]})
mylist = ['a', 'b', 'c', 'd', 'e', 'f']
def get_sublist(sta,end):
return mylist[sta:end+1]
df['col_3'] = df.apply(lambda x: get_sublist(x.col_1, x.col_2), axis=1)
打印输出(df):
ID col_1 col_2 col_3
0 1 0 1 [a, b]
1 2 2 4 [c, d, e]
2 3 3 5 [d, e, f]
如果你的列名包含空格或与现有的dataframe属性共享一个名称，你可以用方括号索引:
df['col_3'] = df.apply(lambda x: f(x['col 1'], x['col 2']), axis=1)
您正在寻找的方法是Series.combine。
然而，在数据类型方面似乎需要多加注意。
在您的示例中，您会(就像我在测试答案时那样)天真地调用
df['col_3'] = df.col_1.combine(df.col_2, func=get_sublist)
但是，这会抛出错误:
ValueError: setting an array element with a sequence.
我最好的猜测是，它似乎期望结果与调用方法的系列(df。col_1这里)。然而，以下工作:
df['col_3'] = df.col_1.astype(object).combine(df.col_2, func=get_sublist)
df
ID col_1 col_2 col_3
0 1 0 1 [a, b]
1 2 2 4 [c, d, e]
2 3 3 5 [d, e, f]
这里有一个更快的解决方案:
def func_1(a,b):
return a + b
df["C"] = func_1(df["A"].to_numpy(),df["B"].to_numpy())
这比df快380倍。从@Aman应用(f，轴=1)，比df['col_3'] = df快310倍。应用(x: f(x。Col_1, x.col_2)， axis=1) from @ajrwhite。
我还添加了一些基准:
结果:
FUNCTIONS TIMINGS GAIN
apply lambda 0.7 x 1
apply 0.56 x 1.25
map 0.3 x 2.3
np.vectorize 0.01 x 70
f3 on Series 0.0026 x 270
f3 on np arrays 0.0018 x 380
f3 numba 0.0018 x 380
简而言之:
使用apply很慢。我们可以非常简单地加快速度，只需要使用一个函数直接操作Pandas系列(或者更好地操作numpy数组)。因为我们将操作Pandas Series或numpy数组，我们将能够向量化操作。该函数将返回一个Pandas Series或numpy数组，我们将其赋值为一个新列。
下面是基准代码:
import timeit
timeit_setup = """
import pandas as pd
import numpy as np
import numba
np.random.seed(0)
# Create a DataFrame of 10000 rows with 2 columns "A" and "B"
# containing integers between 0 and 100
df = pd.DataFrame(np.random.randint(0,10,size=(10000, 2)), columns=["A", "B"])
def f1(a,b):
# Here a and b are the values of column A and B for a specific row: integers
return a + b
def f2(x):
# Here, x is pandas Series, and corresponds to a specific row of the DataFrame
# 0 and 1 are the indexes of columns A and B
return x[0] + x[1]
def f3(a,b):
# Same as f1 but we will pass parameters that will allow vectorization
# Here, A and B will be Pandas Series or numpy arrays
# with df["C"] = f3(df["A"],df["B"]): Pandas Series
# with df["C"] = f3(df["A"].to_numpy(),df["B"].to_numpy()): numpy arrays
return a + b
@numba.njit('int64[:](int64[:], int64[:])')
def f3_numba_vectorize(a,b):
# Here a and b are 2 numpy arrays with dtype int64
# This function must return a numpy array whith dtype int64
return a + b
"""
test_functions = [
'df["C"] = df.apply(lambda row: f1(row["A"], row["B"]), axis=1)',
'df["C"] = df.apply(f2, axis=1)',
'df["C"] = list(map(f3,df["A"],df["B"]))',
'df["C"] = np.vectorize(f3) (df["A"].to_numpy(),df["B"].to_numpy())',
'df["C"] = f3(df["A"],df["B"])',
'df["C"] = f3(df["A"].to_numpy(),df["B"].to_numpy())',
'df["C"] = f3_numba_vectorize(df["A"].to_numpy(),df["B"].to_numpy())'
]
for test_function in test_functions:
print(min(timeit.repeat(setup=timeit_setup, stmt=test_function, repeat=7, number=10)))
输出:
0.7
0.56
0.3
0.01
0.0026
0.0018
0.0018
最后注意:事情可以优化Cython和其他numba技巧。
我要投票支持np。vectorize。它允许你只拍摄x个列，而不处理函数中的数据帧，所以它非常适合你不控制的函数，或者做一些像发送2列和一个常数到一个函数(即col_1, col_2， 'foo')。
import numpy as np
import pandas as pd
df = pd.DataFrame({'ID':['1','2','3'], 'col_1': [0,2,3], 'col_2':[1,4,5]})
mylist = ['a','b','c','d','e','f']
def get_sublist(sta,end):
return mylist[sta:end+1]
#df['col_3'] = df[['col_1','col_2']].apply(get_sublist,axis=1)
# expect above to output df as below
df.loc[:,'col_3'] = np.vectorize(get_sublist, otypes=["O"]) (df['col_1'], df['col_2'])
df
ID col_1 col_2 col_3
0 1 0 1 [a, b]
1 2 2 4 [c, d, e]
2 3 3 5 [d, e, f]
