While Jon Skeet's answer is an excellent advice for everyday's practice with small to moderate number of elements (up to a few millions) it is nevertheless not the fastest approach and not very resource efficient. An obvious drawback is the fact that getting the full difference requires two passes over the data (even three if the elements that are equal are of interest as well). Clearly, this can be avoided by a customized reimplementation of the Except method, but it remains that the creation of a hash set requires a lot of memory and the computation of hashes requires time.
对于非常大的数据集(数十亿个元素),考虑特定的情况通常是有好处的。这里有一些想法可能会给你一些启发:
如果元素可以比较(在实践中几乎总是这样),那么对列表进行排序并应用以下zip方法是值得考虑的:
/// <returns>The elements of the specified (ascendingly) sorted enumerations that are
/// contained only in one of them, together with an indicator,
/// whether the element is contained in the reference enumeration (-1)
/// or in the difference enumeration (+1).</returns>
public static IEnumerable<Tuple<T, int>> FindDifferences<T>(IEnumerable<T> sortedReferenceObjects,
IEnumerable<T> sortedDifferenceObjects, IComparer<T> comparer)
{
var refs = sortedReferenceObjects.GetEnumerator();
var diffs = sortedDifferenceObjects.GetEnumerator();
bool hasNext = refs.MoveNext() && diffs.MoveNext();
while (hasNext)
{
int comparison = comparer.Compare(refs.Current, diffs.Current);
if (comparison == 0)
{
// insert code that emits the current element if equal elements should be kept
hasNext = refs.MoveNext() && diffs.MoveNext();
}
else if (comparison < 0)
{
yield return Tuple.Create(refs.Current, -1);
hasNext = refs.MoveNext();
}
else
{
yield return Tuple.Create(diffs.Current, 1);
hasNext = diffs.MoveNext();
}
}
}
例如,它可以以以下方式使用:
const int N = <Large number>;
const int omit1 = 231567;
const int omit2 = 589932;
IEnumerable<int> numberSequence1 = Enumerable.Range(0, N).Select(i => i < omit1 ? i : i + 1);
IEnumerable<int> numberSequence2 = Enumerable.Range(0, N).Select(i => i < omit2 ? i : i + 1);
var numberDiffs = FindDifferences(numberSequence1, numberSequence2, Comparer<int>.Default);
在我的计算机上对N = 1M进行基准测试,得到以下结果:
Method |
Mean |
Error |
StdDev |
Ratio |
Gen 0 |
Gen 1 |
Gen 2 |
Allocated |
DiffLinq |
115.19 ms |
0.656 ms |
0.582 ms |
1.00 |
2800.0000 |
2800.0000 |
2800.0000 |
67110744 B |
DiffZip |
23.48 ms |
0.018 ms |
0.015 ms |
0.20 |
- |
- |
- |
720 B |
对于N = 100M:
Method |
Mean |
Error |
StdDev |
Ratio |
Gen 0 |
Gen 1 |
Gen 2 |
Allocated |
DiffLinq |
12.146 s |
0.0427 s |
0.0379 s |
1.00 |
13000.0000 |
13000.0000 |
13000.0000 |
8589937032 B |
DiffZip |
2.324 s |
0.0019 s |
0.0018 s |
0.19 |
- |
- |
- |
720 B |
请注意,这个示例当然得益于列表已经排序并且可以非常有效地比较整数的事实。但这正是关键所在:如果你确实有有利的环境,一定要好好利用它们。
A few further comments: The speed of the comparison function is clearly relevant for the overall performance, so it may be beneficial to optimize it. The flexibility to do so is a benefit of the zipping approach. Furthermore, parallelization seems more feasible to me, although by no means easy and maybe not worth the effort and the overhead. Nevertheless, a simple way to speed up the process by roughly a factor of 2, is to split the lists respectively in two halfs (if it can be efficiently done) and compare the parts in parallel, one processing from front to back and the other in reverse order.