I had worked on a project, where my aim was finding Trending Topics from Live Twitter Stream and also doing sentimental analysis on the trending topics (finding if Trending Topic positively/negatively talked about). I've used Storm for handling twitter stream. I've published my report as a blog: http://sayrohan.blogspot.com/2013/06/finding-trending-topics-and-trending.html I've used Total Count and Z-Score for the ranking. The approach that I've used is bit generic, and in the discussion section, I've mentioned that how we can extend the system for non-Twitter Application. Hope the information helps.

我认为你需要注意的关键词是“不正常”。为了确定什么时候“不正常”，你必须知道什么是正常的。也就是说，您将需要历史数据，可以对其求平均值以找出特定查询的正常速率。您可能希望从平均计算中排除不正常的日子，但这同样需要有足够的数据，这样您就知道应该排除哪些日子。

在此基础上，你必须设置一个阈值(我确信这需要实验)，如果有东西超出了阈值，比如搜索量比正常情况多50%，你就可以认为这是一个“趋势”。或者，如果你想像你提到的那样找到“最流行的X”，你只需要根据它们与正常比率的距离(百分比)来排序。

例如，假设你的历史数据告诉你，布兰妮·斯皮尔斯(Britney Spears)通常获得10万次搜索，帕丽斯·希尔顿(Paris Hilton)通常获得5万次搜索。如果有一天她们的搜索量都比平时多了1万次，那么你应该认为帕里斯比布兰妮更“热”，因为她的搜索量比平时多了20%，而布兰妮的搜索量只增加了10%。

天啊，我真不敢相信我刚刚写了一段比较布兰妮·斯皮尔斯和帕丽斯·希尔顿的“性感”的段落。你对我做了什么?

这个问题需要一个z分数或标准分数，它会考虑历史平均值，就像其他人提到的，但也会考虑历史数据的标准差，这使得它比仅仅使用平均值更可靠。

在你的例子中，z分数是通过以下公式计算的，其中趋势是一个比率，如观看次数/天。

z-score = ([current trend] - [average historic trends]) / [standard deviation of historic trends]

当使用z分数时，z分数越高或越低，趋势就越不正常，例如，如果z分数高度正，那么趋势就会异常上升，而如果z分数高度负，则趋势就会异常下降。因此，一旦你计算出所有候选趋势的z分数，最高的10个z分数将与最不正常增长的z分数有关。

有关z分数的更多信息，请参阅维基百科。

Code

from math import sqrt

def zscore(obs, pop):
    # Size of population.
    number = float(len(pop))
    # Average population value.
    avg = sum(pop) / number
    # Standard deviation of population.
    std = sqrt(sum(((c - avg) ** 2) for c in pop) / number)
    # Zscore Calculation.
    return (obs - avg) / std

样例输出

>>> zscore(12, [2, 4, 4, 4, 5, 5, 7, 9])
3.5
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20])
0.0739221270955
>>> zscore(20, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
1.00303599234
>>> zscore(2, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1])
-0.922793112954
>>> zscore(9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0])
1.65291949506

笔记

You can use this method with a sliding window (i.e. last 30 days) if you wish not to take to much history into account, which will make short term trends more pronounced and can cut down on the processing time. You could also use a z-score for values such as change in views from one day to next day to locate the abnormal values for increasing/decreasing views per day. This is like using the slope or derivative of the views per day graph. If you keep track of the current size of the population, the current total of the population, and the current total of x^2 of the population, you don't need to recalculate these values, only update them and hence you only need to keep these values for the history, not each data value. The following code demonstrates this. from math import sqrt class zscore: def __init__(self, pop = []): self.number = float(len(pop)) self.total = sum(pop) self.sqrTotal = sum(x ** 2 for x in pop) def update(self, value): self.number += 1.0 self.total += value self.sqrTotal += value ** 2 def avg(self): return self.total / self.number def std(self): return sqrt((self.sqrTotal / self.number) - self.avg() ** 2) def score(self, obs): return (obs - self.avg()) / self.std() Using this method your work flow would be as follows. For each topic, tag, or page create a floating point field, for the total number of days, sum of views, and sum of views squared in your database. If you have historic data, initialize these fields using that data, otherwise initialize to zero. At the end of each day, calculate the z-score using the day's number of views against the historic data stored in the three database fields. The topics, tags, or pages, with the highest X z-scores are your X "hotest trends" of the day. Finally update each of the 3 fields with the day's value and repeat the process next day.

新成员

Normal z-scores as discussed above do not take into account the order of the data and hence the z-score for an observation of '1' or '9' would have the same magnitude against the sequence [1, 1, 1, 1, 9, 9, 9, 9]. Obviously for trend finding, the most current data should have more weight than older data and hence we want the '1' observation to have a larger magnitude score than the '9' observation. In order to achieve this I propose a floating average z-score. It should be clear that this method is NOT guaranteed to be statistically sound but should be useful for trend finding or similar. The main difference between the standard z-score and the floating average z-score is the use of a floating average to calculate the average population value and the average population value squared. See code for details:

Code

class fazscore:
    def __init__(self, decay, pop = []):
        self.sqrAvg = self.avg = 0
        # The rate at which the historic data's effect will diminish.
        self.decay = decay
        for x in pop: self.update(x)
    def update(self, value):
        # Set initial averages to the first value in the sequence.
        if self.avg == 0 and self.sqrAvg == 0:
            self.avg = float(value)
            self.sqrAvg = float((value ** 2))
        # Calculate the average of the rest of the values using a 
        # floating average.
        else:
            self.avg = self.avg * self.decay + value * (1 - self.decay)
            self.sqrAvg = self.sqrAvg * self.decay + (value ** 2) * (1 - self.decay)
        return self
    def std(self):
        # Somewhat ad-hoc standard deviation calculation.
        return sqrt(self.sqrAvg - self.avg ** 2)
    def score(self, obs):
        if self.std() == 0: return (obs - self.avg) * float("infinity")
        else: return (obs - self.avg) / self.std()

样例输入输出

>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(1)
-1.67770595327
>>> fazscore(0.8, [1, 1, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9]).score(9)
0.596052006642
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(12)
3.46442230724
>>> fazscore(0.9, [2, 4, 4, 4, 5, 5, 7, 9]).score(22)
7.7773245459
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20]).score(20)
-0.24633160155
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(20)
1.1069362749
>>> fazscore(0.9, [21, 22, 19, 18, 17, 22, 20, 20, 1, 2, 3, 1, 2, 1, 0, 1]).score(2)
-0.786764452966
>>> fazscore(0.9, [1, 2, 0, 3, 1, 3, 1, 2, 9, 8, 7, 10, 9, 5, 2, 4, 1, 1, 0]).score(9)
1.82262469243
>>> fazscore(0.8, [40] * 200).score(1)
-inf

更新

正如David Kemp所正确指出的那样，如果给定一系列常数值，然后要求一个与其他值不同的观测值的zscore，那么结果应该是非零的。事实上，返回的值应该是无穷大。我改变了这条线，

if self.std() == 0: return 0

to:

if self.std() == 0: return (obs - self.avg) * float("infinity")

这一更改反映在fazscore解决方案代码中。如果你不想处理无穷大的值，一个可以接受的解决方案是将这一行改为:

if self.std() == 0: return obs - self.avg

你需要一种算法来衡量一个话题的传播速度——换句话说，如果你把它画出来，你想要显示那些以惊人的速度增长的话题。

这是趋势线的一阶导数，将其作为整体计算的加权因素并不难。

正常化

One technique you'll need to do is to normalize all your data. For each topic you are following, keep a very low pass filter that defines that topic's baseline. Now every data point that comes in about that topic should be normalized - subtract its baseline and you'll get ALL of your topics near 0, with spikes above and below the line. You may instead want to divide the signal by its baseline magnitude, which will bring the signal to around 1.0 - this not only brings all signals in line with each other (normalizes the baseline), but also normalizes the spikes. A britney spike is going to be magnitudes larger than someone else's spike, but that doesn't mean you should pay attention to it - the spike may be very small relative to her baseline.

推导出

一旦你标准化了所有的东西，计算出每个主题的斜率。取两个连续的点，测量差值。正的差异呈上升趋势，负的差异呈下降趋势。然后你可以比较归一化的差异，并找出哪些主题与其他主题相比受欢迎程度上升-每个主题都适用于它自己的“正常”，这可能是与其他主题不同的量级。

这确实是解决这个问题的第一步。您还需要使用更高级的技术(主要是上述算法与其他算法的组合，根据您的需要进行加权)，但这应该足以让您开始学习。

关于文章

The article is about topic trending, but it's not about how to calculate what's hot and what's not, it's about how to process the huge amount of information that such an algorithm must process at places like Lycos and Google. The space and time required to give each topic a counter, and find each topic's counter when a search on it goes through is huge. This article is about the challenges one faces when attempting such a task. It does mention the Brittney effect, but it doesn't talk about how to overcome it.

正如Nixuz指出的，这也被称为Z或标准分数。

我想知道在这种情况下是否有可能使用常规的物理加速度公式?

v2-v1/t or dv/dt

我们可以认为v1是每小时的初始点赞/投票/评论数，v2是过去24小时内每小时的当前“速度”?

这更像是一个问题，而不是一个答案，但似乎它可能会起作用。任何加速最快的内容都将成为热门话题……

我相信这并不能解决布兰妮的问题:-)

通常情况下，“嗡嗡声”是通过某种形式的指数/对数衰减机制计算出来的。有关Hacker News、Reddit和其他网站如何简单处理这个问题的概述，请参阅这篇文章。

这并没有完全解决总是受欢迎的事情。你要找的似乎是谷歌的“热门趋势”功能。为此，您可以将当前值除以历史值，然后减去低于某个噪声阈值的值。