如何在整数列表中找到重复项并创建重复项的另一个列表?


当前回答

一个非常简单的解决方案,但是复杂度是O(n*n)。

>>> xs = [1,2,3,4,4,5,5,6,1]
>>> set([x for x in xs if xs.count(x) > 1])
set([1, 4, 5])

其他回答

list2 = [1, 2, 3, 4, 1, 2, 3]
lset = set()
[(lset.add(item), list2.append(item))
 for item in list2 if item not in lset]
print list(lset)
some_list = ['a', 'b', 'c', 'b', 'd', 'm', 'n', 'n']
some_dictionary = {}

for element in some_list:
    if element not in some_dictionary:
       some_dictionary[element] = 1
    else:
        some_dictionary[element] += 1

for key, value in some_dictionary.items():
    if value > 1:
       print(key, end = ' ')

# another way
duplicates = []

for x in some_list:
    if some_list.count(x) > 1 and x not in duplicates:
        duplicates.append(x)

print()
print(duplicates)

来源:这里

还有其他测试。当然要做……

set([x for x in l if l.count(x) > 1])

...代价太大了。使用下一个final方法大约快500倍(数组越长结果越好):

def dups_count_dict(l):
    d = {}

    for item in l:
        if item not in d:
            d[item] = 0

        d[item] += 1

    result_d = {key: val for key, val in d.iteritems() if val > 1}

    return result_d.keys()

只有2个循环,没有非常昂贵的l.count()操作。

下面是一个比较方法的代码。代码如下,输出如下:

dups_count: 13.368s # this is a function which uses l.count()
dups_count_dict: 0.014s # this is a final best function (of the 3 functions)
dups_count_counter: 0.024s # collections.Counter

测试代码:

import numpy as np
from time import time
from collections import Counter

class TimerCounter(object):
    def __init__(self):
        self._time_sum = 0

    def start(self):
        self.time = time()

    def stop(self):
        self._time_sum += time() - self.time

    def get_time_sum(self):
        return self._time_sum


def dups_count(l):
    return set([x for x in l if l.count(x) > 1])


def dups_count_dict(l):
    d = {}

    for item in l:
        if item not in d:
            d[item] = 0

        d[item] += 1

    result_d = {key: val for key, val in d.iteritems() if val > 1}

    return result_d.keys()


def dups_counter(l):
    counter = Counter(l)    

    result_d = {key: val for key, val in counter.iteritems() if val > 1}

    return result_d.keys()



def gen_array():
    np.random.seed(17)
    return list(np.random.randint(0, 5000, 10000))


def assert_equal_results(*results):
    primary_result = results[0]
    other_results = results[1:]

    for other_result in other_results:
        assert set(primary_result) == set(other_result) and len(primary_result) == len(other_result)


if __name__ == '__main__':
    dups_count_time = TimerCounter()
    dups_count_dict_time = TimerCounter()
    dups_count_counter = TimerCounter()

    l = gen_array()

    for i in range(3):
        dups_count_time.start()
        result1 = dups_count(l)
        dups_count_time.stop()

        dups_count_dict_time.start()
        result2 = dups_count_dict(l)
        dups_count_dict_time.stop()

        dups_count_counter.start()
        result3 = dups_counter(l)
        dups_count_counter.stop()

        assert_equal_results(result1, result2, result3)

    print 'dups_count: %.3f' % dups_count_time.get_time_sum()
    print 'dups_count_dict: %.3f' % dups_count_dict_time.get_time_sum()
    print 'dups_count_counter: %.3f' % dups_count_counter.get_time_sum()

尽管它的复杂度是O(n log n),但这似乎有点竞争性,请参阅下面的基准测试。

a = sorted(a)
dupes = list(set(a[::2]) & set(a[1::2]))

排序会把副本放在一起,所以它们都在偶数下标和奇数下标处。唯一值只能在偶数或奇数下标处存在,不能同时存在。所以偶数下标值和奇数下标值的交集就是重复项。

基准测试结果:

这使用了MSeifert的基准测试,但只使用了从接受的答案(georgs)、最慢的解决方案、最快的解决方案(不包括it_duplcopies,因为它不唯一重复)和我的解决方案。否则就太拥挤了,颜色也太相似了。

如果允许修改给定的列表,那么第一行可以是a.sort(),这样会快一些。但是基准会多次重用相同的列表,因此修改它会打乱基准。

显然set(a[::2]).intersection(a[1::2])不会创建第二个集合,而且速度会快一点,但它也会长一点。

第三个接受答案的例子给出了一个错误的答案,并且没有试图给出重复的答案。下面是正确的版本:

number_lst = [1, 1, 2, 3, 5, ...]

seen_set = set()
duplicate_set = set(x for x in number_lst if x in seen_set or seen_set.add(x))
unique_set = seen_set - duplicate_set