有一个表消息,其中包含如下所示的数据:

Id   Name   Other_Columns
-------------------------
1    A       A_data_1
2    A       A_data_2
3    A       A_data_3
4    B       B_data_1
5    B       B_data_2
6    C       C_data_1

如果我按名称从消息组中运行查询select *,我将得到如下结果:

1    A       A_data_1
4    B       B_data_1
6    C       C_data_1

哪个查询将返回以下结果?

3    A       A_data_3
5    B       B_data_2
6    C       C_data_1

也就是说,应该返回每个组中的最后一条记录。

目前,这是我使用的查询:

SELECT
  *
FROM (SELECT
  *
FROM messages
ORDER BY id DESC) AS x
GROUP BY name

但这看起来效率很低。还有其他方法可以达到同样的效果吗?


当前回答

UPD: 2017-03-31, MySQL 5.7.5版本默认启用了ONLY_FULL_GROUP_BY开关(因此,不确定的GROUP by查询被禁用)。此外,他们更新了GROUP BY实现,即使禁用了开关,解决方案也可能不再像预期的那样工作。我们需要检查一下。

Bill Karwin的上述解决方案在组内的项目计数相当小时工作得很好,但当组相当大时查询性能就会变差,因为该解决方案只需要n*n/2 + n/2个is NULL比较。

我在一个包含18684446行和1182个组的InnoDB表上进行了测试。该表包含功能测试的测试结果,并将(test_id, request_id)作为主键。因此,test_id是一个组,我正在为每个test_id搜索最后的request_id。

Bill的解决方案已经在我的dell e4310上运行了几个小时,我不知道它什么时候会完成,即使它在覆盖索引上运行(因此在EXPLAIN中使用索引)。

基于同样的想法,我有一些其他的解决方案:

if the underlying index is BTREE index (which is usually the case), the largest (group_id, item_value) pair is the last value within each group_id, that is the first for each group_id if we walk through the index in descending order; if we read the values which are covered by an index, the values are read in the order of the index; each index implicitly contains primary key columns appended to that (that is the primary key is in the coverage index). In solutions below I operate directly on the primary key, in you case, you will just need to add primary key columns in the result. in many cases it is much cheaper to collect the required row ids in the required order in a subquery and join the result of the subquery on the id. Since for each row in the subquery result MySQL will need a single fetch based on primary key, the subquery will be put first in the join and the rows will be output in the order of the ids in the subquery (if we omit explicit ORDER BY for the join)

MySQL使用索引的3种方式是一篇了解一些细节的好文章。

解决方案1

这个是非常快的,在我的18M+行上大约需要0.8秒:

SELECT test_id, MAX(request_id) AS request_id
FROM testresults
GROUP BY test_id DESC;

如果你想改变顺序为ASC,把它放在一个子查询中,只返回id,并使用它作为子查询连接到其他列:

SELECT test_id, request_id
FROM (
    SELECT test_id, MAX(request_id) AS request_id
    FROM testresults
    GROUP BY test_id DESC) as ids
ORDER BY test_id;

在我的数据上,这个大约需要1,2秒。

解决方案2

下面是另一个解决方案,对我的表来说大约需要19秒:

SELECT test_id, request_id
FROM testresults, (SELECT @group:=NULL) as init
WHERE IF(IFNULL(@group, -1)=@group:=test_id, 0, 1)
ORDER BY test_id DESC, request_id DESC

它还按降序返回测试。它的速度要慢得多,因为它做了一个完整的索引扫描,但它在这里给你一个想法,如何为每个组输出最多N行。

查询的缺点是查询缓存不能缓存其结果。

其他回答

显然,有很多不同的方法得到相同的结果,你的问题似乎是什么是一个有效的方法得到最后的结果在每组MySQL。如果你正在处理大量的数据,并且假设你正在使用InnoDB,即使是最新版本的MySQL(比如5.7.21和8.0.4-rc),那么可能没有一个有效的方法来做到这一点。

有时我们需要对超过6000万行的表执行此操作。

对于这些示例,我将使用只有大约150万行的数据,其中查询需要为数据中的所有组找到结果。在我们的实际情况中,我们经常需要返回大约2000个组的数据(假设不需要检查很多数据)。

我将使用以下表格:

CREATE TABLE temperature(
  id INT UNSIGNED NOT NULL AUTO_INCREMENT, 
  groupID INT UNSIGNED NOT NULL, 
  recordedTimestamp TIMESTAMP NOT NULL, 
  recordedValue INT NOT NULL,
  INDEX groupIndex(groupID, recordedTimestamp), 
  PRIMARY KEY (id)
);

CREATE TEMPORARY TABLE selected_group(id INT UNSIGNED NOT NULL, PRIMARY KEY(id)); 

温度表由大约150万条随机记录和100个不同的组填充。 selected_group由这100个组填充(在我们的例子中,所有组通常小于20%)。

由于该数据是随机的,这意味着多行可以具有相同的recordedTimestamps。我们想要的是获得所有选中的组的列表,这些组按groupID顺序排列,每个组的最后一个recordedTimestamp,如果同一个组有多个这样的匹配行,那么这些行的最后一个匹配id。

如果假设MySQL有一个last()函数,它从一个特殊的ORDER BY子句中返回最后一行的值,那么我们可以简单地这样做:

SELECT 
  last(t1.id) AS id, 
  t1.groupID, 
  last(t1.recordedTimestamp) AS recordedTimestamp, 
  last(t1.recordedValue) AS recordedValue
FROM selected_group g
INNER JOIN temperature t1 ON t1.groupID = g.id
ORDER BY t1.recordedTimestamp, t1.id
GROUP BY t1.groupID;

which would only need to examine a few 100 rows in this case as it doesn't use any of the normal GROUP BY functions. This would execute in 0 seconds and hence be highly efficient. Note that normally in MySQL we would see an ORDER BY clause following the GROUP BY clause however this ORDER BY clause is used to determine the ORDER for the last() function, if it was after the GROUP BY then it would be ordering the GROUPS. If no GROUP BY clause is present then the last values will be the same in all of the returned rows.

但是MySQL没有这个,所以让我们看看它有什么不同的想法,并证明这些都不是有效的。

示例1

SELECT t1.id, t1.groupID, t1.recordedTimestamp, t1.recordedValue
FROM selected_group g
INNER JOIN temperature t1 ON t1.id = (
  SELECT t2.id
  FROM temperature t2 
  WHERE t2.groupID = g.id
  ORDER BY t2.recordedTimestamp DESC, t2.id DESC
  LIMIT 1
);

这检查了3,009,254行,在5.7.21上花费了0.859秒,在8.0.4-rc上花费了稍长的时间

示例2

SELECT t1.id, t1.groupID, t1.recordedTimestamp, t1.recordedValue 
FROM temperature t1
INNER JOIN ( 
  SELECT max(t2.id) AS id   
  FROM temperature t2
  INNER JOIN (
    SELECT t3.groupID, max(t3.recordedTimestamp) AS recordedTimestamp
    FROM selected_group g
    INNER JOIN temperature t3 ON t3.groupID = g.id
    GROUP BY t3.groupID
  ) t4 ON t4.groupID = t2.groupID AND t4.recordedTimestamp = t2.recordedTimestamp
  GROUP BY t2.groupID
) t5 ON t5.id = t1.id;

这检查了1,505,331行,在5.7.21上花费了约1.25秒,在8.0.4-rc上花费了稍长时间

示例3

SELECT t1.id, t1.groupID, t1.recordedTimestamp, t1.recordedValue 
FROM temperature t1
WHERE t1.id IN ( 
  SELECT max(t2.id) AS id   
  FROM temperature t2
  INNER JOIN (
    SELECT t3.groupID, max(t3.recordedTimestamp) AS recordedTimestamp
    FROM selected_group g
    INNER JOIN temperature t3 ON t3.groupID = g.id
    GROUP BY t3.groupID
  ) t4 ON t4.groupID = t2.groupID AND t4.recordedTimestamp = t2.recordedTimestamp
  GROUP BY t2.groupID
)
ORDER BY t1.groupID;

这检查了3,009,685行,在5.7.21上花费了约1.95秒,在8.0.4-rc上花费了稍长时间

示例4

SELECT t1.id, t1.groupID, t1.recordedTimestamp, t1.recordedValue
FROM selected_group g
INNER JOIN temperature t1 ON t1.id = (
  SELECT max(t2.id)
  FROM temperature t2 
  WHERE t2.groupID = g.id AND t2.recordedTimestamp = (
      SELECT max(t3.recordedTimestamp)
      FROM temperature t3 
      WHERE t3.groupID = g.id
    )
);

这检查了6,137,810行,在5.7.21上花费了约2.2秒,在8.0.4-rc上花费了稍长时间

示例5

SELECT t1.id, t1.groupID, t1.recordedTimestamp, t1.recordedValue
FROM (
  SELECT 
    t2.id, 
    t2.groupID, 
    t2.recordedTimestamp, 
    t2.recordedValue, 
    row_number() OVER (
      PARTITION BY t2.groupID ORDER BY t2.recordedTimestamp DESC, t2.id DESC
    ) AS rowNumber
  FROM selected_group g 
  INNER JOIN temperature t2 ON t2.groupID = g.id
) t1 WHERE t1.rowNumber = 1;

这检查了6,017,808行,在8.0.4-rc上花费了约4.2秒

例子6

SELECT t1.id, t1.groupID, t1.recordedTimestamp, t1.recordedValue 
FROM (
  SELECT 
    last_value(t2.id) OVER w AS id, 
    t2.groupID, 
    last_value(t2.recordedTimestamp) OVER w AS recordedTimestamp, 
    last_value(t2.recordedValue) OVER w AS recordedValue
  FROM selected_group g
  INNER JOIN temperature t2 ON t2.groupID = g.id
  WINDOW w AS (
    PARTITION BY t2.groupID 
    ORDER BY t2.recordedTimestamp, t2.id 
    RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
  )
) t1
GROUP BY t1.groupID;

这检查了6,017,908行,在8.0.4-rc上花费了约17.5秒

例7

SELECT t1.id, t1.groupID, t1.recordedTimestamp, t1.recordedValue 
FROM selected_group g
INNER JOIN temperature t1 ON t1.groupID = g.id
LEFT JOIN temperature t2 
  ON t2.groupID = g.id 
  AND (
    t2.recordedTimestamp > t1.recordedTimestamp 
    OR (t2.recordedTimestamp = t1.recordedTimestamp AND t2.id > t1.id)
  )
WHERE t2.id IS NULL
ORDER BY t1.groupID;

这只花了很长时间,所以我不得不杀了它。

我还没有测试大DB,但我认为这可能比连接表更快:

SELECT *, Max(Id) FROM messages GROUP BY Name

一个相当快的方法如下。

SELECT * 
FROM messages a
WHERE Id = (SELECT MAX(Id) FROM messages WHERE a.Name = Name)

结果

Id  Name    Other_Columns
3   A   A_data_3
5   B   B_data_2
6   C   C_data_1

是什么:

select *, max(id) from messages group by name 

我已经在sqlite上测试了它,它返回所有列和所有名称的最大id值。

使用子查询返回正确的分组,因为您已经完成了一半。

试试这个:

select
    a.*
from
    messages a
    inner join 
        (select name, max(id) as maxid from messages group by name) as b on
        a.id = b.maxid

如果它不是id,你想要的最大值:

select
    a.*
from
    messages a
    inner join 
        (select name, max(other_col) as other_col 
         from messages group by name) as b on
        a.name = b.name
        and a.other_col = b.other_col

通过这种方式,可以避免在子查询中进行相关子查询和/或排序,这往往非常缓慢/低效。