比较两个庞大(>50.000项)的最快(和最少资源密集型)的方法是什么,从而得到如下所示的两个列表:

在第一个列表中出现但在第二个列表中没有出现的项目 出现在第二个列表中但不在第一个列表中的项目

目前,我正在使用列表或IReadOnlyCollection,并在linq查询中解决这个问题:

var list1 = list.Where(i => !list2.Contains(i)).ToList();
var list2 = list2.Where(i => !list.Contains(i)).ToList();

但这并不像我想的那样好。 有什么想法使这更快和更少的资源密集,因为我需要处理很多列表?


当前回答

可列举的。SequenceEqual方法 根据相等比较器确定两个序列是否相等。 MS.Docs

Enumerable.SequenceEqual(list1, list2);

这适用于所有基本数据类型。如果你需要在自定义对象上使用它,你需要实现IEqualityComparer

定义方法以支持相等的对象比较。

IEqualityComparer接口 定义方法以支持相等的对象比较。 MS.Docs for IEqualityComparer

其他回答

这是你能找到的最好的解决办法

var list3 = list1.Where(l => list2.ToList().Contains(l));

第一种方法:

if (list1 != null && list2 != null && list1.Select(x => list2.SingleOrDefault(y => y.propertyToCompare == x.propertyToCompare && y.anotherPropertyToCompare == x.anotherPropertyToCompare) != null).All(x => true))
   return true;

第二种方法,如果你对重复值没有问题:

if (list1 != null && list2 != null && list1.Select(x => list2.Any(y => y.propertyToCompare == x.propertyToCompare && y.anotherPropertyToCompare == x.anotherPropertyToCompare)).All(x => true))
   return true;

While Jon Skeet's answer is an excellent advice for everyday's practice with small to moderate number of elements (up to a few millions) it is nevertheless not the fastest approach and not very resource efficient. An obvious drawback is the fact that getting the full difference requires two passes over the data (even three if the elements that are equal are of interest as well). Clearly, this can be avoided by a customized reimplementation of the Except method, but it remains that the creation of a hash set requires a lot of memory and the computation of hashes requires time.

对于非常大的数据集(数十亿个元素),考虑特定的情况通常是有好处的。这里有一些想法可能会给你一些启发: 如果元素可以比较(在实践中几乎总是这样),那么对列表进行排序并应用以下zip方法是值得考虑的:

/// <returns>The elements of the specified (ascendingly) sorted enumerations that are
/// contained only in one of them, together with an indicator,
/// whether the element is contained in the reference enumeration (-1)
/// or in the difference enumeration (+1).</returns>
public static IEnumerable<Tuple<T, int>> FindDifferences<T>(IEnumerable<T> sortedReferenceObjects,
    IEnumerable<T> sortedDifferenceObjects, IComparer<T> comparer)
{
    var refs  = sortedReferenceObjects.GetEnumerator();
    var diffs = sortedDifferenceObjects.GetEnumerator();
    bool hasNext = refs.MoveNext() && diffs.MoveNext();
    while (hasNext)
    {
        int comparison = comparer.Compare(refs.Current, diffs.Current);
        if (comparison == 0)
        {
            // insert code that emits the current element if equal elements should be kept
            hasNext = refs.MoveNext() && diffs.MoveNext();

        }
        else if (comparison < 0)
        {
            yield return Tuple.Create(refs.Current, -1);
            hasNext = refs.MoveNext();
        }
        else
        {
            yield return Tuple.Create(diffs.Current, 1);
            hasNext = diffs.MoveNext();
        }
    }
}

例如,它可以以以下方式使用:

const int N = <Large number>;
const int omit1 = 231567;
const int omit2 = 589932;
IEnumerable<int> numberSequence1 = Enumerable.Range(0, N).Select(i => i < omit1 ? i : i + 1);
IEnumerable<int> numberSequence2 = Enumerable.Range(0, N).Select(i => i < omit2 ? i : i + 1);
var numberDiffs = FindDifferences(numberSequence1, numberSequence2, Comparer<int>.Default);

在我的计算机上对N = 1M进行基准测试,得到以下结果:

Method Mean Error StdDev Ratio Gen 0 Gen 1 Gen 2 Allocated
DiffLinq 115.19 ms 0.656 ms 0.582 ms 1.00 2800.0000 2800.0000 2800.0000 67110744 B
DiffZip 23.48 ms 0.018 ms 0.015 ms 0.20 - - - 720 B

对于N = 100M:

Method Mean Error StdDev Ratio Gen 0 Gen 1 Gen 2 Allocated
DiffLinq 12.146 s 0.0427 s 0.0379 s 1.00 13000.0000 13000.0000 13000.0000 8589937032 B
DiffZip 2.324 s 0.0019 s 0.0018 s 0.19 - - - 720 B

请注意,这个示例当然得益于列表已经排序并且可以非常有效地比较整数的事实。但这正是关键所在:如果你确实有有利的环境,一定要好好利用它们。

A few further comments: The speed of the comparison function is clearly relevant for the overall performance, so it may be beneficial to optimize it. The flexibility to do so is a benefit of the zipping approach. Furthermore, parallelization seems more feasible to me, although by no means easy and maybe not worth the effort and the overhead. Nevertheless, a simple way to speed up the process by roughly a factor of 2, is to split the lists respectively in two halfs (if it can be efficiently done) and compare the parts in parallel, one processing from front to back and the other in reverse order.

不是针对这个问题,但是这里有一些代码来比较相等和不相等的列表!相同的对象:

public class EquatableList<T> : List<T>, IEquatable<EquatableList<T>> where    T : IEquatable<T>

/// <summary>
/// True, if this contains element with equal property-values
/// </summary>
/// <param name="element">element of Type T</param>
/// <returns>True, if this contains element</returns>
public new Boolean Contains(T element)
{
    return this.Any(t => t.Equals(element));
}

/// <summary>
/// True, if list is equal to this
/// </summary>
/// <param name="list">list</param>
/// <returns>True, if instance equals list</returns>
public Boolean Equals(EquatableList<T> list)
{
    if (list == null) return false;
    return this.All(list.Contains) && list.All(this.Contains);
}

我做了比较两个列表的泛型函数。

 public static class ListTools
{
    public enum RecordUpdateStatus
    {
        Added = 1,
        Updated = 2,
        Deleted = 3
    }


    public class UpdateStatu<T>
    {
        public T CurrentValue { get; set; }
        public RecordUpdateStatus UpdateStatus { get; set; }
    }

    public static List<UpdateStatu<T>> CompareList<T>(List<T> currentList, List<T> inList, string uniqPropertyName)
    {
        var res = new List<UpdateStatu<T>>();

        res.AddRange(inList.Where(a => !currentList.Any(x => x.GetType().GetProperty(uniqPropertyName).GetValue(x)?.ToString().ToLower() == a.GetType().GetProperty(uniqPropertyName).GetValue(a)?.ToString().ToLower()))
            .Select(a => new UpdateStatu<T>
            {
                CurrentValue = a,
                UpdateStatus = RecordUpdateStatus.Added,
            }));

        res.AddRange(currentList.Where(a => !inList.Any(x => x.GetType().GetProperty(uniqPropertyName).GetValue(x)?.ToString().ToLower() == a.GetType().GetProperty(uniqPropertyName).GetValue(a)?.ToString().ToLower()))
            .Select(a => new UpdateStatu<T>
            {
                CurrentValue = a,
                UpdateStatus = RecordUpdateStatus.Deleted,
            }));


        res.AddRange(currentList.Where(a => inList.Any(x => x.GetType().GetProperty(uniqPropertyName).GetValue(x)?.ToString().ToLower() == a.GetType().GetProperty(uniqPropertyName).GetValue(a)?.ToString().ToLower()))
         .Select(a => new UpdateStatu<T>
         {
             CurrentValue = a,
             UpdateStatus = RecordUpdateStatus.Updated,
         }));

        return res;
    }

}