使用filterfalse而不使用lambda-expression

When using functions like filter or filterfalse and similar from itertools you can usually save performance by avoiding lambda-expressions and using already existing functions. Instances of list and set defines a __contains__-method to use for containment checks. The in-operator calls this method under the hood, so using x in l2 can be replaced by l2.__contains__(x). Usually this replacement is not really prettier but in this specific case it allows us to gain better performance than using a lambda-expression, when used in combination with filterfalse:

>>> from itertools import filterfalse
>>> l1 = [1, 2, 6, 8]
>>> l2 = [2, 3, 5, 8]
>>> list(filterfalse(l2.__contains__, l1))
[1, 6]

Filterfalse创建一个迭代器，该迭代器生成的所有元素在用作12.2 .__contains__的参数时返回false。

Sets有一个更快的__contains__实现，所以更好的是:

>>> from itertools import filterfalse
>>> l1 = [1, 2, 6, 8]
>>> l2 = set([2, 3, 5, 8])
>>> list(filterfalse(l2.__contains__, l1))
[1, 6]

性能

使用列表:

$  python3 -m timeit -s "from itertools import filterfalse; l1 = [1,2,6,8]; l2 = set([2,3,5,8]);" "list(filterfalse(l2.__contains__, l1))"
500000 loops, best of 5: 522 nsec per loop

使用设置:

$ python3 -m timeit -s "from itertools import filterfalse; l1 = [1,2,6,8]; l2 = set([2,3,5,8]);" "list(filterfalse(l2.__contains__, l1))"
1000000 loops, best of 5: 359 nsec per loop

如果你想要那种行为，集合方法是最好的。如果您不想删除列表l1中仅在l2中存在过一次的元素的所有实例，那么这些set操作将导致错误的结果。假设你在l1中有重复的元素，甚至在l2中也有重复的元素，并且想要两个列表l1 - l2的实际差值，同时保持其余元素的顺序:

l1 = [1, 2, 3, 4, 5, 5, 6, 5, 5, 2]
l2 = [1, 2, 2, 5]
_ = [l1.remove(item) for item in l2 if item in l1] # discard return value
print(l1) # [3, 4, 5, 6, 5, 5]

注意，这将明显比设置操作慢，只在用例需要时使用它 如果你不想修改原来的列表-先创建一个列表的副本

扩展Donut的答案和这里的其他答案，通过使用生成器推导式而不是列表推导式，以及使用集合数据结构(因为in操作符在列表上是O(n)，而在集合上是O(1))，您可以得到更好的结果。

这里有一个函数适合你:

def filter_list(full_list, excludes):
    s = set(excludes)
    return (x for x in full_list if x not in s)

结果将是一个可迭代对象，它将惰性地获取过滤后的列表。如果你需要一个真正的列表对象(例如，如果你需要对结果执行len())，那么你可以很容易地像这样构建一个列表:

filtered_list = list(filter_list(full_list, excludes))

使用Set推导式{x for x in l2}或Set (l2)来获取Set，然后使用List推导式来获取List

l2set = set(l2)
l3 = [x for x in l1 if x not in l2set]

基准测试代码:

import time

l1 = list(range(1000*10 * 3))
l2 = list(range(1000*10 * 2))

l2set = {x for x in l2}

tic = time.time()
l3 = [x for x in l1 if x not in l2set]
toc = time.time()
diffset = toc-tic
print(diffset)

tic = time.time()
l3 = [x for x in l1 if x not in l2]
toc = time.time()
difflist = toc-tic
print(difflist)

print("speedup %fx"%(difflist/diffset))

基准测试结果:

0.0015058517456054688
3.968189239501953
speedup 2635.179227x    

通过利用字典的有序属性来维持顺序(Python 3.7+)

注意:Python 3.6中字典的参考实现按照插入顺序维护键，但规范不保证这一点。对于3.7及更高版本，添加了这个保证。

字典的键作为一种集合;重复项被隐式过滤掉，由于散列，查找是高效的。因此，我们可以通过使用l1作为键来构建字典，然后删除出现在l2中的任何键来实现“set difference”。这维持了秩序并使用了一种快速的算法，但会产生相当数量的常量开销。

d = dict.fromkeys(l1)
for i in l2:
    try:
        del d[i]
    except KeyError:
        pass
l3 = list(d.keys())

使用filterfalse而不使用lambda-expression

When using functions like filter or filterfalse and similar from itertools you can usually save performance by avoiding lambda-expressions and using already existing functions. Instances of list and set defines a __contains__-method to use for containment checks. The in-operator calls this method under the hood, so using x in l2 can be replaced by l2.__contains__(x). Usually this replacement is not really prettier but in this specific case it allows us to gain better performance than using a lambda-expression, when used in combination with filterfalse:

>>> from itertools import filterfalse
>>> l1 = [1, 2, 6, 8]
>>> l2 = [2, 3, 5, 8]
>>> list(filterfalse(l2.__contains__, l1))
[1, 6]

Filterfalse创建一个迭代器，该迭代器生成的所有元素在用作12.2 .__contains__的参数时返回false。

Sets有一个更快的__contains__实现，所以更好的是:

>>> from itertools import filterfalse
>>> l1 = [1, 2, 6, 8]
>>> l2 = set([2, 3, 5, 8])
>>> list(filterfalse(l2.__contains__, l1))
[1, 6]

性能

使用列表:

$  python3 -m timeit -s "from itertools import filterfalse; l1 = [1,2,6,8]; l2 = set([2,3,5,8]);" "list(filterfalse(l2.__contains__, l1))"
500000 loops, best of 5: 522 nsec per loop

使用设置:

$ python3 -m timeit -s "from itertools import filterfalse; l1 = [1,2,6,8]; l2 = set([2,3,5,8]);" "list(filterfalse(l2.__contains__, l1))"
1000000 loops, best of 5: 359 nsec per loop