I had worked on a project, where my aim was finding Trending Topics from Live Twitter Stream and also doing sentimental analysis on the trending topics (finding if Trending Topic positively/negatively talked about). I've used Storm for handling twitter stream. I've published my report as a blog: http://sayrohan.blogspot.com/2013/06/finding-trending-topics-and-trending.html I've used Total Count and Z-Score for the ranking. The approach that I've used is bit generic, and in the discussion section, I've mentioned that how we can extend the system for non-Twitter Application. Hope the information helps.

我们的想法是跟踪这些东西，并注意到它们什么时候比自己的基线跳得明显。

因此，对于具有超过某个阈值的查询，跟踪每个查询，当它更改为其历史值的某个值时(例如几乎是其历史值的两倍)，那么它将成为一个新的热门趋势。

您可以使用对数概率比来比较当前日期与上个月或去年。这在统计上是合理的(假设你的事件不是正态分布，这是从你的问题中假设的)。

只需按logLR排序所有的术语，并选择前10名。

public static void main(String... args) {
    TermBag today = ...
    TermBag lastYear = ...
    for (String each: today.allTerms()) {
        System.out.println(logLikelihoodRatio(today, lastYear, each) + "\t" + each);
    }
} 

public static double logLikelihoodRatio(TermBag t1, TermBag t2, String term) {
    double k1 = t1.occurrences(term); 
    double k2 = t2.occurrences(term); 
    double n1 = t1.size(); 
    double n2 = t2.size(); 
    double p1 = k1 / n1;
    double p2 = k2 / n2;
    double p = (k1 + k2) / (n1 + n2);
    double logLR = 2*(logL(p1,k1,n1) + logL(p2,k2,n2) - logL(p,k1,n1) - logL(p,k2,n2));
    if (p1 < p2) logLR *= -1;
    return logLR;
}

private static double logL(double p, double k, double n) {
    return (k == 0 ? 0 : k * Math.log(p)) + ((n - k) == 0 ? 0 : (n - k) * Math.log(1 - p));
}

PS, TermBag是单词的无序集合。为每个文档创建一袋术语。数一下单词的出现次数。然后，occurrences方法返回给定单词的出现次数，size方法返回单词的总数。最好以某种方式规范化单词，通常toLowerCase就足够了。当然，在上面的示例中，您将创建一个包含当前所有查询的文档，以及一个包含去年所有查询的文档。

这个问题需要一个z分数或标准分数，它会考虑历史平均值，就像其他人提到的，但也会考虑历史数据的标准差，这使得它比仅仅使用平均值更可靠。

在你的例子中，z分数是通过以下公式计算的，其中趋势是一个比率，如观看次数/天。

z-score = ([current trend] - [average historic trends]) / [standard deviation of historic trends]

当使用z分数时，z分数越高或越低，趋势就越不正常，例如，如果z分数高度正，那么趋势就会异常上升，而如果z分数高度负，则趋势就会异常下降。因此，一旦你计算出所有候选趋势的z分数，最高的10个z分数将与最不正常增长的z分数有关。

有关z分数的更多信息，请参阅维基百科。

Code

from math import sqrt

def zscore(obs, pop):
    # Size of population.
    number = float(len(pop))
    # Average population value.
    avg = sum(pop) / number
    # Standard deviation of population.
    std = sqrt(sum(((c - avg) ** 2) for c in pop) / number)
    # Zscore Calculation.
    return (obs - avg) / std

样例输出

>>> zscore(12, [2, 4, 4, 4, 5, 5, 7, 9])
3.5
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20])
0.0739221270955
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
1.00303599234
>>> zscore(2, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
-0.922793112954
>>> zscore(9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0])
1.65291949506

笔记

You can use this method with a sliding window (i.e. last 30 days) if you wish not to take to much history into account, which will make short term trends more pronounced and can cut down on the processing time. You could also use a z-score for values such as change in views from one day to next day to locate the abnormal values for increasing/decreasing views per day. This is like using the slope or derivative of the views per day graph. If you keep track of the current size of the population, the current total of the population, and the current total of x^2 of the population, you don't need to recalculate these values, only update them and hence you only need to keep these values for the history, not each data value. The following code demonstrates this. from math import sqrt class zscore: def __init__(self, pop = []): self.number = float(len(pop)) self.total = sum(pop) self.sqrTotal = sum(x ** 2 for x in pop) def update(self, value): self.number += 1.0 self.total += value self.sqrTotal += value ** 2 def avg(self): return self.total / self.number def std(self): return sqrt((self.sqrTotal / self.number) - self.avg() ** 2) def score(self, obs): return (obs - self.avg()) / self.std() Using this method your work flow would be as follows. For each topic, tag, or page create a floating point field, for the total number of days, sum of views, and sum of views squared in your database. If you have historic data, initialize these fields using that data, otherwise initialize to zero. At the end of each day, calculate the z-score using the day's number of views against the historic data stored in the three database fields. The topics, tags, or pages, with the highest X z-scores are your X "hotest trends" of the day. Finally update each of the 3 fields with the day's value and repeat the process next day.

新成员

Normal z-scores as discussed above do not take into account the order of the data and hence the z-score for an observation of '1' or '9' would have the same magnitude against the sequence [1, 1, 1, 1, 9, 9, 9, 9]. Obviously for trend finding, the most current data should have more weight than older data and hence we want the '1' observation to have a larger magnitude score than the '9' observation. In order to achieve this I propose a floating average z-score. It should be clear that this method is NOT guaranteed to be statistically sound but should be useful for trend finding or similar. The main difference between the standard z-score and the floating average z-score is the use of a floating average to calculate the average population value and the average population value squared. See code for details:

Code

class fazscore:
    def __init__(self, decay, pop = []):
        self.sqrAvg = self.avg = 0
        # The rate at which the historic data's effect will diminish.
        self.decay = decay
        for x in pop: self.update(x)
    def update(self, value):
        # Set initial averages to the first value in the sequence.
        if self.avg == 0 and self.sqrAvg == 0:
            self.avg = float(value)
            self.sqrAvg = float((value ** 2))
        # Calculate the average of the rest of the values using a 
        # floating average.
        else:
            self.avg = self.avg * self.decay + value * (1 - self.decay)
            self.sqrAvg = self.sqrAvg * self.decay + (value ** 2) * (1 - self.decay)
        return self
    def std(self):
        # Somewhat ad-hoc standard deviation calculation.
        return sqrt(self.sqrAvg - self.avg ** 2)
    def score(self, obs):
        if self.std() == 0: return (obs - self.avg) * float("infinity")
        else: return (obs - self.avg) / self.std()

样例输入输出

>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(1)
-1.67770595327
>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(9)
0.596052006642
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(12)
3.46442230724
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(22)
7.7773245459
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20]).score(20)
-0.24633160155
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(20)
1.1069362749
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(2)
-0.786764452966
>>> fazscore(0.9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0]).score(9)
1.82262469243
>>> fazscore(0.8, [40] * 200).score(1)
-inf

更新

正如David Kemp所正确指出的那样，如果给定一系列常数值，然后要求一个与其他值不同的观测值的zscore，那么结果应该是非零的。事实上，返回的值应该是无穷大。我改变了这条线，

if self.std() == 0: return 0

to:

if self.std() == 0: return (obs - self.avg) * float("infinity")

这一更改反映在fazscore解决方案代码中。如果你不想处理无穷大的值，一个可以接受的解决方案是将这一行改为:

if self.std() == 0: return obs - self.avg

通常情况下，“嗡嗡声”是通过某种形式的指数/对数衰减机制计算出来的。有关Hacker News、Reddit和其他网站如何简单处理这个问题的概述，请参阅这篇文章。

这并没有完全解决总是受欢迎的事情。你要找的似乎是谷歌的“热门趋势”功能。为此，您可以将当前值除以历史值，然后减去低于某个噪声阈值的值。

我想知道在这种情况下是否有可能使用常规的物理加速度公式?

v2-v1/t or dv/dt

我们可以认为v1是每小时的初始点赞/投票/评论数，v2是过去24小时内每小时的当前“速度”?

这更像是一个问题，而不是一个答案，但似乎它可能会起作用。任何加速最快的内容都将成为热门话题……

我相信这并不能解决布兰妮的问题:-)