比较两个庞大(>50.000项)的最快(和最少资源密集型)的方法是什么,从而得到如下所示的两个列表:

在第一个列表中出现但在第二个列表中没有出现的项目 出现在第二个列表中但不在第一个列表中的项目

目前,我正在使用列表或IReadOnlyCollection,并在linq查询中解决这个问题:

var list1 = list.Where(i => !list2.Contains(i)).ToList();
var list2 = list2.Where(i => !list.Contains(i)).ToList();

但这并不像我想的那样好。 有什么想法使这更快和更少的资源密集,因为我需要处理很多列表?


当前回答

使用除外:

var firstNotSecond = list1.Except(list2).ToList();
var secondNotFirst = list2.Except(list1).ToList();

我怀疑有一些方法实际上会比这个稍微快一点,但即使是这个方法也会比O(N * M)方法快得多。

如果你想把它们结合起来,你可以用上面的方法创建一个方法,然后创建一个return语句:

return !firstNotSecond.Any() && !secondNotFirst.Any();

需要注意的一点是,问题中的原始代码和这里的解决方案之间的结果存在差异:在我的代码中,只在一个列表中出现的任何重复元素将只报告一次,而在原始代码中出现的次数与它们相同。

例如,对于[1,2,2,2,3]和[1]的列表,原始代码中的“元素在list1但不是list2”结果将是[2,2,2,3]。在我的代码中,它就是[2,3]。在许多情况下,这不是一个问题,但这是值得注意的。

其他回答

不是针对这个问题,但是这里有一些代码来比较相等和不相等的列表!相同的对象:

public class EquatableList<T> : List<T>, IEquatable<EquatableList<T>> where    T : IEquatable<T>

/// <summary>
/// True, if this contains element with equal property-values
/// </summary>
/// <param name="element">element of Type T</param>
/// <returns>True, if this contains element</returns>
public new Boolean Contains(T element)
{
    return this.Any(t => t.Equals(element));
}

/// <summary>
/// True, if list is equal to this
/// </summary>
/// <param name="list">list</param>
/// <returns>True, if instance equals list</returns>
public Boolean Equals(EquatableList<T> list)
{
    if (list == null) return false;
    return this.All(list.Contains) && list.All(this.Contains);
}

我做了比较两个列表的泛型函数。

 public static class ListTools
{
    public enum RecordUpdateStatus
    {
        Added = 1,
        Updated = 2,
        Deleted = 3
    }


    public class UpdateStatu<T>
    {
        public T CurrentValue { get; set; }
        public RecordUpdateStatus UpdateStatus { get; set; }
    }

    public static List<UpdateStatu<T>> CompareList<T>(List<T> currentList, List<T> inList, string uniqPropertyName)
    {
        var res = new List<UpdateStatu<T>>();

        res.AddRange(inList.Where(a => !currentList.Any(x => x.GetType().GetProperty(uniqPropertyName).GetValue(x)?.ToString().ToLower() == a.GetType().GetProperty(uniqPropertyName).GetValue(a)?.ToString().ToLower()))
            .Select(a => new UpdateStatu<T>
            {
                CurrentValue = a,
                UpdateStatus = RecordUpdateStatus.Added,
            }));

        res.AddRange(currentList.Where(a => !inList.Any(x => x.GetType().GetProperty(uniqPropertyName).GetValue(x)?.ToString().ToLower() == a.GetType().GetProperty(uniqPropertyName).GetValue(a)?.ToString().ToLower()))
            .Select(a => new UpdateStatu<T>
            {
                CurrentValue = a,
                UpdateStatus = RecordUpdateStatus.Deleted,
            }));


        res.AddRange(currentList.Where(a => inList.Any(x => x.GetType().GetProperty(uniqPropertyName).GetValue(x)?.ToString().ToLower() == a.GetType().GetProperty(uniqPropertyName).GetValue(a)?.ToString().ToLower()))
         .Select(a => new UpdateStatu<T>
         {
             CurrentValue = a,
             UpdateStatus = RecordUpdateStatus.Updated,
         }));

        return res;
    }

}

更有效的方法是使用Enumerable。除了:

var inListButNotInList2 = list.Except(list2);
var inList2ButNotInList = list2.Except(list);

该方法是通过使用延迟执行实现的。这意味着你可以这样写:

var first10 = inListButNotInList2.Take(10);

它也很有效,因为它在内部使用Set<T>来比较对象。它的工作原理是首先从第二个序列中收集所有不同的值,然后将第一个序列的结果流式传输,检查它们是否之前没有出现过。

While Jon Skeet's answer is an excellent advice for everyday's practice with small to moderate number of elements (up to a few millions) it is nevertheless not the fastest approach and not very resource efficient. An obvious drawback is the fact that getting the full difference requires two passes over the data (even three if the elements that are equal are of interest as well). Clearly, this can be avoided by a customized reimplementation of the Except method, but it remains that the creation of a hash set requires a lot of memory and the computation of hashes requires time.

对于非常大的数据集(数十亿个元素),考虑特定的情况通常是有好处的。这里有一些想法可能会给你一些启发: 如果元素可以比较(在实践中几乎总是这样),那么对列表进行排序并应用以下zip方法是值得考虑的:

/// <returns>The elements of the specified (ascendingly) sorted enumerations that are
/// contained only in one of them, together with an indicator,
/// whether the element is contained in the reference enumeration (-1)
/// or in the difference enumeration (+1).</returns>
public static IEnumerable<Tuple<T, int>> FindDifferences<T>(IEnumerable<T> sortedReferenceObjects,
    IEnumerable<T> sortedDifferenceObjects, IComparer<T> comparer)
{
    var refs  = sortedReferenceObjects.GetEnumerator();
    var diffs = sortedDifferenceObjects.GetEnumerator();
    bool hasNext = refs.MoveNext() && diffs.MoveNext();
    while (hasNext)
    {
        int comparison = comparer.Compare(refs.Current, diffs.Current);
        if (comparison == 0)
        {
            // insert code that emits the current element if equal elements should be kept
            hasNext = refs.MoveNext() && diffs.MoveNext();

        }
        else if (comparison < 0)
        {
            yield return Tuple.Create(refs.Current, -1);
            hasNext = refs.MoveNext();
        }
        else
        {
            yield return Tuple.Create(diffs.Current, 1);
            hasNext = diffs.MoveNext();
        }
    }
}

例如,它可以以以下方式使用:

const int N = <Large number>;
const int omit1 = 231567;
const int omit2 = 589932;
IEnumerable<int> numberSequence1 = Enumerable.Range(0, N).Select(i => i < omit1 ? i : i + 1);
IEnumerable<int> numberSequence2 = Enumerable.Range(0, N).Select(i => i < omit2 ? i : i + 1);
var numberDiffs = FindDifferences(numberSequence1, numberSequence2, Comparer<int>.Default);

在我的计算机上对N = 1M进行基准测试,得到以下结果:

Method Mean Error StdDev Ratio Gen 0 Gen 1 Gen 2 Allocated
DiffLinq 115.19 ms 0.656 ms 0.582 ms 1.00 2800.0000 2800.0000 2800.0000 67110744 B
DiffZip 23.48 ms 0.018 ms 0.015 ms 0.20 - - - 720 B

对于N = 100M:

Method Mean Error StdDev Ratio Gen 0 Gen 1 Gen 2 Allocated
DiffLinq 12.146 s 0.0427 s 0.0379 s 1.00 13000.0000 13000.0000 13000.0000 8589937032 B
DiffZip 2.324 s 0.0019 s 0.0018 s 0.19 - - - 720 B

请注意,这个示例当然得益于列表已经排序并且可以非常有效地比较整数的事实。但这正是关键所在:如果你确实有有利的环境,一定要好好利用它们。

A few further comments: The speed of the comparison function is clearly relevant for the overall performance, so it may be beneficial to optimize it. The flexibility to do so is a benefit of the zipping approach. Furthermore, parallelization seems more feasible to me, although by no means easy and maybe not worth the effort and the overhead. Nevertheless, a simple way to speed up the process by roughly a factor of 2, is to split the lists respectively in two halfs (if it can be efficiently done) and compare the parts in parallel, one processing from front to back and the other in reverse order.

我认为这是一个简单易行的方法来逐个元素比较两个列表

x=[1,2,3,5,4,8,7,11,12,45,96,25]
y=[2,4,5,6,8,7,88,9,6,55,44,23]

tmp = []


for i in range(len(x)) and range(len(y)):
    if x[i]>y[i]:
        tmp.append(1)
    else:
        tmp.append(0)
print(tmp)