要根据一组现有字段汇总多个字段，请使用下面的示例:(根据您的需求替换$1、$2、$3、$4)

cat file

US|A|1000|2000
US|B|1000|2000
US|C|1000|2000
UK|1|1000|2000
UK|1|1000|2000
UK|1|1000|2000

awk 'BEGIN { FS=OFS=SUBSEP="|"}{arr[$1,$2]+=$3+$4 }END {for (i in arr) print i,arr[i]}' file

US|A|3000
US|B|3000
US|C|3000
UK|1|9000

典型的解决方案是另一位受访者提到的:

sort | uniq -c

它比用Perl或awk编写的代码更短、更简洁。

You write that you don't want to use sort, because the data's size is larger than the machine's main memory size. Don't underestimate the implementation quality of the Unix sort command. Sort was used to handle very large volumes of data (think the original AT&T's billing data) on machines with 128k (that's 131,072 bytes) of memory (PDP-11). When sort encounters more data than a preset limit (often tuned close to the size of the machine's main memory) it sorts the data it has read in main memory and writes it into a temporary file. It then repeats the action with the next chunks of data. Finally, it performs a merge sort on those intermediate files. This allows sort to work on data many times larger than the machine's main memory.

如果顺序不重要，排序可以省略

uniq -c <source_file>

or

echo "$list" | uniq -c

如果源列表是一个变量

cat ip_addresses | sort | uniq -c | sort -nr | awk '{print $2 " " $1}'

这个命令将提供您想要的输出

似乎您必须使用大量代码在bash中模拟哈希以获得线性行为，或者坚持使用二次超线性版本。

在这些版本中，saua的解决方案是最好的(也是最简单的):

sort -n ip_addresses.txt | uniq -c

我找到了http://unix.derkeiler.com/Newsgroups/comp.unix.shell/2005-11/0118.html。但它丑得要命……

awk + sort(带版本排序标志)的组合可能是最快的(如果你的环境有awk的话):

echo "${input...}" |

{m,g}awk '{ __[$+_]++ } END { for(_ in __) { print "",+__[_],_ } }' FS='^$' OFS='\t' | 

gsort -t$'\t' -k 3,3 -V

只有后GROUP-BY汇总行被发送到排序实用程序——与毫无理由地对输入行进行预先排序相比，这是一种系统密集型排序。

对于小输入，例如少于1000行左右，只需直接排序|uniq -c它。

    3   10.0.10.1
    1   10.0.10.2
    1   10.0.10.3