大多数其他解决方案计算重复。如果你真的需要分组键值对，试试这个:

以下是我的示例数据:

find . | xargs md5sum
fe4ab8e15432161f452e345ff30c68b0 a.txt
30c68b02161e15435ff52e34f4fe4ab8 b.txt
30c68b02161e15435ff52e34f4fe4ab8 c.txt
fe4ab8e15432161f452e345ff30c68b0 d.txt
fe4ab8e15432161f452e345ff30c68b0 e.txt

这将打印按md5校验和分组的键值对。

cat table.txt | awk '{print $1}' | sort | uniq  | xargs -i grep {} table.txt
30c68b02161e15435ff52e34f4fe4ab8 b.txt
30c68b02161e15435ff52e34f4fe4ab8 c.txt
fe4ab8e15432161f452e345ff30c68b0 a.txt
fe4ab8e15432161f452e345ff30c68b0 d.txt
fe4ab8e15432161f452e345ff30c68b0 e.txt

我知道你在Bash中寻找一些东西，但如果其他人可能在Python中寻找一些东西，你可能会考虑这样做:

mySet = set()
for line in open("ip_address_file.txt"):
     line = line.rstrip()
     mySet.add(line)

由于集合中的值在默认情况下是唯一的，而Python在这方面非常擅长，因此您可能会在这里赢得一些东西。我还没有测试代码，所以它可能有漏洞，但这可能会让你明白。如果你想要计数出现的次数，使用字典而不是集合很容易实现。

编辑: 我不擅长阅读，所以我答错了。这里有一个字典片段，可以计算发生的次数。

mydict = {}
for line in open("ip_address_file.txt"):
    line = line.rstrip()
    if line in mydict:
        mydict[line] += 1
    else:
        mydict[line] = 1

字典mydict现在保存一个唯一IP的列表作为键，它们出现的次数作为值。

典型的解决方案是另一位受访者提到的:

sort | uniq -c

它比用Perl或awk编写的代码更短、更简洁。

You write that you don't want to use sort, because the data's size is larger than the machine's main memory size. Don't underestimate the implementation quality of the Unix sort command. Sort was used to handle very large volumes of data (think the original AT&T's billing data) on machines with 128k (that's 131,072 bytes) of memory (PDP-11). When sort encounters more data than a preset limit (often tuned close to the size of the machine's main memory) it sorts the data it has read in main memory and writes it into a temporary file. It then repeats the action with the next chunks of data. Finally, it performs a merge sort on those intermediate files. This allows sort to work on data many times larger than the machine's main memory.

我会这样做:

perl -e 'while (<>) {chop; $h{$_}++;} for $k (keys %h) {print "$k $h{$k}\n";}' ip_addresses

但uniq可能适合你。

cat ip_addresses | sort | uniq -c | sort -nr | awk '{print $2 " " $1}'

这个命令将提供您想要的输出

如果顺序不重要，排序可以省略

uniq -c <source_file>

or

echo "$list" | uniq -c

如果源列表是一个变量