您可以使用对数概率比来比较当前日期与上个月或去年。这在统计上是合理的(假设你的事件不是正态分布，这是从你的问题中假设的)。

只需按logLR排序所有的术语，并选择前10名。

public static void main(String... args) {
    TermBag today = ...
    TermBag lastYear = ...
    for (String each: today.allTerms()) {
        System.out.println(logLikelihoodRatio(today, lastYear, each) + "\t" + each);
    }
} 

public static double logLikelihoodRatio(TermBag t1, TermBag t2, String term) {
    double k1 = t1.occurrences(term); 
    double k2 = t2.occurrences(term); 
    double n1 = t1.size(); 
    double n2 = t2.size(); 
    double p1 = k1 / n1;
    double p2 = k2 / n2;
    double p = (k1 + k2) / (n1 + n2);
    double logLR = 2*(logL(p1,k1,n1) + logL(p2,k2,n2) - logL(p,k1,n1) - logL(p,k2,n2));
    if (p1 < p2) logLR *= -1;
    return logLR;
}

private static double logL(double p, double k, double n) {
    return (k == 0 ? 0 : k * Math.log(p)) + ((n - k) == 0 ? 0 : (n - k) * Math.log(1 - p));
}

PS, TermBag是单词的无序集合。为每个文档创建一袋术语。数一下单词的出现次数。然后，occurrences方法返回给定单词的出现次数，size方法返回单词的总数。最好以某种方式规范化单词，通常toLowerCase就足够了。当然，在上面的示例中，您将创建一个包含当前所有查询的文档，以及一个包含去年所有查询的文档。

您可以使用对数概率比来比较当前日期与上个月或去年。这在统计上是合理的(假设你的事件不是正态分布，这是从你的问题中假设的)。

只需按logLR排序所有的术语，并选择前10名。

public static void main(String... args) {
    TermBag today = ...
    TermBag lastYear = ...
    for (String each: today.allTerms()) {
        System.out.println(logLikelihoodRatio(today, lastYear, each) + "\t" + each);
    }
} 

public static double logLikelihoodRatio(TermBag t1, TermBag t2, String term) {
    double k1 = t1.occurrences(term); 
    double k2 = t2.occurrences(term); 
    double n1 = t1.size(); 
    double n2 = t2.size(); 
    double p1 = k1 / n1;
    double p2 = k2 / n2;
    double p = (k1 + k2) / (n1 + n2);
    double logLR = 2*(logL(p1,k1,n1) + logL(p2,k2,n2) - logL(p,k1,n1) - logL(p,k2,n2));
    if (p1 < p2) logLR *= -1;
    return logLR;
}

private static double logL(double p, double k, double n) {
    return (k == 0 ? 0 : k * Math.log(p)) + ((n - k) == 0 ? 0 : (n - k) * Math.log(1 - p));
}

PS, TermBag是单词的无序集合。为每个文档创建一袋术语。数一下单词的出现次数。然后，occurrences方法返回给定单词的出现次数，size方法返回单词的总数。最好以某种方式规范化单词，通常toLowerCase就足够了。当然，在上面的示例中，您将创建一个包含当前所有查询的文档，以及一个包含去年所有查询的文档。

如果你只是看推文或状态信息来获取你的主题，你会遇到很多噪音。即使你删除了所有的停止词。获得更好的主题候选子集的一种方法是只关注共享URL的推文/消息，并从这些网页的标题中获得关键字。并且确保你也应用了POS标记来获得名词+名词短语。

网页的标题通常更具有描述性，包含描述页面内容的单词。此外，分享网页通常与分享突发新闻相关(例如，如果像迈克尔·杰克逊这样的名人去世了，你会有很多人分享关于他去世的文章)。

我做过实验，只从标题中选取热门关键词，然后在所有状态信息中计算这些关键词的总数，它们确实消除了很多干扰。如果你这样做，你不需要一个复杂的算法，只是做一个简单的关键字频率排序，你已经完成了一半。

这个问题需要一个z分数或标准分数，它会考虑历史平均值，就像其他人提到的，但也会考虑历史数据的标准差，这使得它比仅仅使用平均值更可靠。

在你的例子中，z分数是通过以下公式计算的，其中趋势是一个比率，如观看次数/天。

z-score = ([current trend] - [average historic trends]) / [standard deviation of historic trends]

当使用z分数时，z分数越高或越低，趋势就越不正常，例如，如果z分数高度正，那么趋势就会异常上升，而如果z分数高度负，则趋势就会异常下降。因此，一旦你计算出所有候选趋势的z分数，最高的10个z分数将与最不正常增长的z分数有关。

有关z分数的更多信息，请参阅维基百科。

Code

from math import sqrt

def zscore(obs, pop):
    # Size of population.
    number = float(len(pop))
    # Average population value.
    avg = sum(pop) / number
    # Standard deviation of population.
    std = sqrt(sum(((c - avg) ** 2) for c in pop) / number)
    # Zscore Calculation.
    return (obs - avg) / std

样例输出

>>> zscore(12, [2, 4, 4, 4, 5, 5, 7, 9])
3.5
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20])
0.0739221270955
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
1.00303599234
>>> zscore(2, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
-0.922793112954
>>> zscore(9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0])
1.65291949506

笔记

You can use this method with a sliding window (i.e. last 30 days) if you wish not to take to much history into account, which will make short term trends more pronounced and can cut down on the processing time. You could also use a z-score for values such as change in views from one day to next day to locate the abnormal values for increasing/decreasing views per day. This is like using the slope or derivative of the views per day graph. If you keep track of the current size of the population, the current total of the population, and the current total of x^2 of the population, you don't need to recalculate these values, only update them and hence you only need to keep these values for the history, not each data value. The following code demonstrates this. from math import sqrt class zscore: def __init__(self, pop = []): self.number = float(len(pop)) self.total = sum(pop) self.sqrTotal = sum(x ** 2 for x in pop) def update(self, value): self.number += 1.0 self.total += value self.sqrTotal += value ** 2 def avg(self): return self.total / self.number def std(self): return sqrt((self.sqrTotal / self.number) - self.avg() ** 2) def score(self, obs): return (obs - self.avg()) / self.std() Using this method your work flow would be as follows. For each topic, tag, or page create a floating point field, for the total number of days, sum of views, and sum of views squared in your database. If you have historic data, initialize these fields using that data, otherwise initialize to zero. At the end of each day, calculate the z-score using the day's number of views against the historic data stored in the three database fields. The topics, tags, or pages, with the highest X z-scores are your X "hotest trends" of the day. Finally update each of the 3 fields with the day's value and repeat the process next day.

新成员

Normal z-scores as discussed above do not take into account the order of the data and hence the z-score for an observation of '1' or '9' would have the same magnitude against the sequence [1, 1, 1, 1, 9, 9, 9, 9]. Obviously for trend finding, the most current data should have more weight than older data and hence we want the '1' observation to have a larger magnitude score than the '9' observation. In order to achieve this I propose a floating average z-score. It should be clear that this method is NOT guaranteed to be statistically sound but should be useful for trend finding or similar. The main difference between the standard z-score and the floating average z-score is the use of a floating average to calculate the average population value and the average population value squared. See code for details:

Code

class fazscore:
    def __init__(self, decay, pop = []):
        self.sqrAvg = self.avg = 0
        # The rate at which the historic data's effect will diminish.
        self.decay = decay
        for x in pop: self.update(x)
    def update(self, value):
        # Set initial averages to the first value in the sequence.
        if self.avg == 0 and self.sqrAvg == 0:
            self.avg = float(value)
            self.sqrAvg = float((value ** 2))
        # Calculate the average of the rest of the values using a 
        # floating average.
        else:
            self.avg = self.avg * self.decay + value * (1 - self.decay)
            self.sqrAvg = self.sqrAvg * self.decay + (value ** 2) * (1 - self.decay)
        return self
    def std(self):
        # Somewhat ad-hoc standard deviation calculation.
        return sqrt(self.sqrAvg - self.avg ** 2)
    def score(self, obs):
        if self.std() == 0: return (obs - self.avg) * float("infinity")
        else: return (obs - self.avg) / self.std()

样例输入输出

>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(1)
-1.67770595327
>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(9)
0.596052006642
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(12)
3.46442230724
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(22)
7.7773245459
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20]).score(20)
-0.24633160155
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(20)
1.1069362749
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(2)
-0.786764452966
>>> fazscore(0.9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0]).score(9)
1.82262469243
>>> fazscore(0.8, [40] * 200).score(1)
-inf

更新

正如David Kemp所正确指出的那样，如果给定一系列常数值，然后要求一个与其他值不同的观测值的zscore，那么结果应该是非零的。事实上，返回的值应该是无穷大。我改变了这条线，

if self.std() == 0: return 0

to:

if self.std() == 0: return (obs - self.avg) * float("infinity")

这一更改反映在fazscore解决方案代码中。如果你不想处理无穷大的值，一个可以接受的解决方案是将这一行改为:

if self.std() == 0: return obs - self.avg

我认为你需要注意的关键词是“不正常”。为了确定什么时候“不正常”，你必须知道什么是正常的。也就是说，您将需要历史数据，可以对其求平均值以找出特定查询的正常速率。您可能希望从平均计算中排除不正常的日子，但这同样需要有足够的数据，这样您就知道应该排除哪些日子。

在此基础上，你必须设置一个阈值(我确信这需要实验)，如果有东西超出了阈值，比如搜索量比正常情况多50%，你就可以认为这是一个“趋势”。或者，如果你想像你提到的那样找到“最流行的X”，你只需要根据它们与正常比率的距离(百分比)来排序。

例如，假设你的历史数据告诉你，布兰妮·斯皮尔斯(Britney Spears)通常获得10万次搜索，帕丽斯·希尔顿(Paris Hilton)通常获得5万次搜索。如果有一天她们的搜索量都比平时多了1万次，那么你应该认为帕里斯比布兰妮更“热”，因为她的搜索量比平时多了20%，而布兰妮的搜索量只增加了10%。

天啊，我真不敢相信我刚刚写了一段比较布兰妮·斯皮尔斯和帕丽斯·希尔顿的“性感”的段落。你对我做了什么?

也许一个简单的话题频率梯度就能起作用——大的正梯度=快速增长的受欢迎程度。

最简单的方法是将每天的搜索次数归位，这样你就有了

searches = [ 10, 7, 14, 8, 9, 12, 55, 104, 100 ]

然后看看它每天有多少变化:

hot_factor = [ b-a for a, b in zip(searches[:-1], searches[1:]) ]
# hot_factor is [ -3, 7, -6, 1, 3, 43, 49, -4 ]

只要应用某种阈值，那么那些增加了> 50的日子就被认为是“热”的。如果你愿意，你也可以让它变得更复杂。不是绝对差异，而是相对差异，所以从100到150被认为是热的，但从1000到1050不是。或者是考虑到不止一天的趋势的更复杂的梯度。