你需要一种算法来衡量一个话题的传播速度——换句话说，如果你把它画出来，你想要显示那些以惊人的速度增长的话题。

这是趋势线的一阶导数，将其作为整体计算的加权因素并不难。

正常化

One technique you'll need to do is to normalize all your data. For each topic you are following, keep a very low pass filter that defines that topic's baseline. Now every data point that comes in about that topic should be normalized - subtract its baseline and you'll get ALL of your topics near 0, with spikes above and below the line. You may instead want to divide the signal by its baseline magnitude, which will bring the signal to around 1.0 - this not only brings all signals in line with each other (normalizes the baseline), but also normalizes the spikes. A britney spike is going to be magnitudes larger than someone else's spike, but that doesn't mean you should pay attention to it - the spike may be very small relative to her baseline.

推导出

一旦你标准化了所有的东西，计算出每个主题的斜率。取两个连续的点，测量差值。正的差异呈上升趋势，负的差异呈下降趋势。然后你可以比较归一化的差异，并找出哪些主题与其他主题相比受欢迎程度上升-每个主题都适用于它自己的“正常”，这可能是与其他主题不同的量级。

这确实是解决这个问题的第一步。您还需要使用更高级的技术(主要是上述算法与其他算法的组合，根据您的需要进行加权)，但这应该足以让您开始学习。

关于文章

The article is about topic trending, but it's not about how to calculate what's hot and what's not, it's about how to process the huge amount of information that such an algorithm must process at places like Lycos and Google. The space and time required to give each topic a counter, and find each topic's counter when a search on it goes through is huge. This article is about the challenges one faces when attempting such a task. It does mention the Brittney effect, but it doesn't talk about how to overcome it.

正如Nixuz指出的，这也被称为Z或标准分数。

如果你只是看推文或状态信息来获取你的主题，你会遇到很多噪音。即使你删除了所有的停止词。获得更好的主题候选子集的一种方法是只关注共享URL的推文/消息，并从这些网页的标题中获得关键字。并且确保你也应用了POS标记来获得名词+名词短语。

网页的标题通常更具有描述性，包含描述页面内容的单词。此外，分享网页通常与分享突发新闻相关(例如，如果像迈克尔·杰克逊这样的名人去世了，你会有很多人分享关于他去世的文章)。

我做过实验，只从标题中选取热门关键词，然后在所有状态信息中计算这些关键词的总数，它们确实消除了很多干扰。如果你这样做，你不需要一个复杂的算法，只是做一个简单的关键字频率排序，你已经完成了一半。

查德·伯奇和亚当·戴维斯的观点是正确的，你必须回顾过去，建立一个基线。你的问题，从措辞上看，表明你只想查看过去24小时的数据，这并不完全正确。

为数据提供一些内存而不必查询大量历史数据的一种方法是使用指数移动平均。这样做的好处是，您可以每个周期更新一次，然后刷新所有旧数据，因此您只需要记住一个值。所以如果你的周期是一天，你必须为每个主题维护一个“每日平均”属性，你可以通过:

a_n = a_(n-1)*b + c_n*(1-b)

其中a_n是第n天的移动平均值，b是0到1之间的某个常数(越接近1，内存越长)，c_n是第n天的点击次数。美妙的是，如果你在第n天结束时执行更新，你可以刷新c_n和a_(n-1)。

需要注意的是，初始时它对a的初始值很敏感。

EDIT

如果这有助于可视化这个方法，取n = 5, a_0 = 1, b = .9。

假设新的值是5,0,0,1,4:

a_0 = 1
c_1 = 5 : a_1 = .9*1 + .1*5 = 1.4
c_2 = 0 : a_2 = .9*1.4 + .1*0 = 1.26
c_3 = 0 : a_3 = .9*1.26 + .1*0 = 1.134
c_4 = 1 : a_4 = .9*1.134 + .1*1 = 1.1206
c_5 = 4 : a_5 = .9*1.1206 + .1*5 = 1.40854

看起来不太像平均值，不是吗?请注意，即使我们的下一个输入是5，该值仍然接近1。这是怎么呢如果你展开计算，你会得到:

a_n = (1-b)*c_n + (1-b)*b*c_(n-1) + (1-b)*b^2*c_(n-2) + ... + (leftover weight)*a_0

我说的剩余重量是什么意思?在任何平均值中，所有的权重都必须加为1。如果n是无穷大，那么。可以一直延续下去，那么所有权值的和都是1。但如果n相对较小，原始输入就会有相当大的权重。

如果你研究了上面的公式，你应该意识到关于这个用法的一些事情:

所有数据永远都对平均值有所贡献。实际上，有一个点的贡献是非常非常小的。 最近的值比旧值贡献更大。 b越高，新值越不重要，旧值越重要。然而，b越高，就需要越多的数据来冲淡a的初值。

我认为前两个特点正是你要找的。为了给你一个简单的想法，这是一个python实现(减去所有的数据库交互):

>>> class EMA(object):
...  def __init__(self, base, decay):
...   self.val = base
...   self.decay = decay
...   print self.val
...  def update(self, value):
...   self.val = self.val*self.decay + (1-self.decay)*value
...   print self.val
... 
>>> a = EMA(1, .9)
1
>>> a.update(10)
1.9
>>> a.update(10)
2.71
>>> a.update(10)
3.439
>>> a.update(10)
4.0951
>>> a.update(10)
4.68559
>>> a.update(10)
5.217031
>>> a.update(10)
5.6953279
>>> a.update(10)
6.12579511
>>> a.update(10)
6.513215599
>>> a.update(10)
6.8618940391
>>> a.update(10)
7.17570463519

这个问题需要一个z分数或标准分数，它会考虑历史平均值，就像其他人提到的，但也会考虑历史数据的标准差，这使得它比仅仅使用平均值更可靠。

在你的例子中，z分数是通过以下公式计算的，其中趋势是一个比率，如观看次数/天。

z-score = ([current trend] - [average historic trends]) / [standard deviation of historic trends]

当使用z分数时，z分数越高或越低，趋势就越不正常，例如，如果z分数高度正，那么趋势就会异常上升，而如果z分数高度负，则趋势就会异常下降。因此，一旦你计算出所有候选趋势的z分数，最高的10个z分数将与最不正常增长的z分数有关。

有关z分数的更多信息，请参阅维基百科。

Code

from math import sqrt

def zscore(obs, pop):
    # Size of population.
    number = float(len(pop))
    # Average population value.
    avg = sum(pop) / number
    # Standard deviation of population.
    std = sqrt(sum(((c - avg) ** 2) for c in pop) / number)
    # Zscore Calculation.
    return (obs - avg) / std

样例输出

>>> zscore(12, [2, 4, 4, 4, 5, 5, 7, 9])
3.5
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20])
0.0739221270955
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
1.00303599234
>>> zscore(2, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
-0.922793112954
>>> zscore(9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0])
1.65291949506

笔记

You can use this method with a sliding window (i.e. last 30 days) if you wish not to take to much history into account, which will make short term trends more pronounced and can cut down on the processing time. You could also use a z-score for values such as change in views from one day to next day to locate the abnormal values for increasing/decreasing views per day. This is like using the slope or derivative of the views per day graph. If you keep track of the current size of the population, the current total of the population, and the current total of x^2 of the population, you don't need to recalculate these values, only update them and hence you only need to keep these values for the history, not each data value. The following code demonstrates this. from math import sqrt class zscore: def __init__(self, pop = []): self.number = float(len(pop)) self.total = sum(pop) self.sqrTotal = sum(x ** 2 for x in pop) def update(self, value): self.number += 1.0 self.total += value self.sqrTotal += value ** 2 def avg(self): return self.total / self.number def std(self): return sqrt((self.sqrTotal / self.number) - self.avg() ** 2) def score(self, obs): return (obs - self.avg()) / self.std() Using this method your work flow would be as follows. For each topic, tag, or page create a floating point field, for the total number of days, sum of views, and sum of views squared in your database. If you have historic data, initialize these fields using that data, otherwise initialize to zero. At the end of each day, calculate the z-score using the day's number of views against the historic data stored in the three database fields. The topics, tags, or pages, with the highest X z-scores are your X "hotest trends" of the day. Finally update each of the 3 fields with the day's value and repeat the process next day.

新成员

Normal z-scores as discussed above do not take into account the order of the data and hence the z-score for an observation of '1' or '9' would have the same magnitude against the sequence [1, 1, 1, 1, 9, 9, 9, 9]. Obviously for trend finding, the most current data should have more weight than older data and hence we want the '1' observation to have a larger magnitude score than the '9' observation. In order to achieve this I propose a floating average z-score. It should be clear that this method is NOT guaranteed to be statistically sound but should be useful for trend finding or similar. The main difference between the standard z-score and the floating average z-score is the use of a floating average to calculate the average population value and the average population value squared. See code for details:

Code

class fazscore:
    def __init__(self, decay, pop = []):
        self.sqrAvg = self.avg = 0
        # The rate at which the historic data's effect will diminish.
        self.decay = decay
        for x in pop: self.update(x)
    def update(self, value):
        # Set initial averages to the first value in the sequence.
        if self.avg == 0 and self.sqrAvg == 0:
            self.avg = float(value)
            self.sqrAvg = float((value ** 2))
        # Calculate the average of the rest of the values using a 
        # floating average.
        else:
            self.avg = self.avg * self.decay + value * (1 - self.decay)
            self.sqrAvg = self.sqrAvg * self.decay + (value ** 2) * (1 - self.decay)
        return self
    def std(self):
        # Somewhat ad-hoc standard deviation calculation.
        return sqrt(self.sqrAvg - self.avg ** 2)
    def score(self, obs):
        if self.std() == 0: return (obs - self.avg) * float("infinity")
        else: return (obs - self.avg) / self.std()

样例输入输出

>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(1)
-1.67770595327
>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(9)
0.596052006642
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(12)
3.46442230724
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(22)
7.7773245459
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20]).score(20)
-0.24633160155
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(20)
1.1069362749
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(2)
-0.786764452966
>>> fazscore(0.9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0]).score(9)
1.82262469243
>>> fazscore(0.8, [40] * 200).score(1)
-inf

更新

正如David Kemp所正确指出的那样，如果给定一系列常数值，然后要求一个与其他值不同的观测值的zscore，那么结果应该是非零的。事实上，返回的值应该是无穷大。我改变了这条线，

if self.std() == 0: return 0

to:

if self.std() == 0: return (obs - self.avg) * float("infinity")

这一更改反映在fazscore解决方案代码中。如果你不想处理无穷大的值，一个可以接受的解决方案是将这一行改为:

if self.std() == 0: return obs - self.avg

我认为你需要注意的关键词是“不正常”。为了确定什么时候“不正常”，你必须知道什么是正常的。也就是说，您将需要历史数据，可以对其求平均值以找出特定查询的正常速率。您可能希望从平均计算中排除不正常的日子，但这同样需要有足够的数据，这样您就知道应该排除哪些日子。

在此基础上，你必须设置一个阈值(我确信这需要实验)，如果有东西超出了阈值，比如搜索量比正常情况多50%，你就可以认为这是一个“趋势”。或者，如果你想像你提到的那样找到“最流行的X”，你只需要根据它们与正常比率的距离(百分比)来排序。

例如，假设你的历史数据告诉你，布兰妮·斯皮尔斯(Britney Spears)通常获得10万次搜索，帕丽斯·希尔顿(Paris Hilton)通常获得5万次搜索。如果有一天她们的搜索量都比平时多了1万次，那么你应该认为帕里斯比布兰妮更“热”，因为她的搜索量比平时多了20%，而布兰妮的搜索量只增加了10%。

天啊，我真不敢相信我刚刚写了一段比较布兰妮·斯皮尔斯和帕丽斯·希尔顿的“性感”的段落。你对我做了什么?

你需要一种算法来衡量一个话题的传播速度——换句话说，如果你把它画出来，你想要显示那些以惊人的速度增长的话题。

这是趋势线的一阶导数，将其作为整体计算的加权因素并不难。

正常化

One technique you'll need to do is to normalize all your data. For each topic you are following, keep a very low pass filter that defines that topic's baseline. Now every data point that comes in about that topic should be normalized - subtract its baseline and you'll get ALL of your topics near 0, with spikes above and below the line. You may instead want to divide the signal by its baseline magnitude, which will bring the signal to around 1.0 - this not only brings all signals in line with each other (normalizes the baseline), but also normalizes the spikes. A britney spike is going to be magnitudes larger than someone else's spike, but that doesn't mean you should pay attention to it - the spike may be very small relative to her baseline.

推导出

一旦你标准化了所有的东西，计算出每个主题的斜率。取两个连续的点，测量差值。正的差异呈上升趋势，负的差异呈下降趋势。然后你可以比较归一化的差异，并找出哪些主题与其他主题相比受欢迎程度上升-每个主题都适用于它自己的“正常”，这可能是与其他主题不同的量级。

这确实是解决这个问题的第一步。您还需要使用更高级的技术(主要是上述算法与其他算法的组合，根据您的需要进行加权)，但这应该足以让您开始学习。

关于文章

The article is about topic trending, but it's not about how to calculate what's hot and what's not, it's about how to process the huge amount of information that such an algorithm must process at places like Lycos and Google. The space and time required to give each topic a counter, and find each topic's counter when a search on it goes through is huge. This article is about the challenges one faces when attempting such a task. It does mention the Brittney effect, but it doesn't talk about how to overcome it.

正如Nixuz指出的，这也被称为Z或标准分数。