我是第一次接触像RabbitMQ这样的消息代理,我们可以使用它为像芹菜这样的调度系统创建任务/消息队列。

现在问题来了:

我可以在PostgreSQL中创建一个表,它可以附加新的任务,并被消费程序(如芹菜)消费。 为什么我要为RabbitMQ这样一个全新的技术呢?

现在,我认为伸缩性不是答案,因为我们的数据库(如PostgreSQL)可以在分布式环境中工作。

我在谷歌上搜索了数据库对特定问题造成的问题,我发现:

轮询使数据库繁忙且性能低下 锁定表->再次低性能 数百万行任务—又是>,轮询的性能很低

现在,RabbitMQ或其他类似的消息代理是如何解决这些问题的呢?

而且,我发现它遵循的是AMQP协议。这有什么好呢?

Redis也可以用作消息代理吗?我发现它更类似于Memcached而不是RabbitMQ。

请解释一下!


当前回答

Rabbit's queues reside in memory and will therefore be much faster than implementing this in a database. A (good)dedicated message queue should also provide essential queuing related features such as throttling/flow control, and the ability to choose different routing algorithms, to name a couple(rabbit provides these and more). Depending on the size of your project, you may also want the message passing component separate from your database, so that if one component experiences heavy load, it need not hinder the other's operation.

关于你提到的问题:

polling keeping the database busy and low performing: Using Rabbitmq, producers can push updates to consumers which is far more performant than polling. Data is simply sent to the consumer when it needs to be, eliminating the need for wasteful checks. locking of the table -> again low performing: There is no table to lock :P millions of rows of task -> again polling is low performing: As mentioned above, Rabbitmq will operate faster as it resides RAM, and provides flow control. If needed, it can also use the disk to temporarily store messages if it runs out of RAM. After 2.0, Rabbit has significantly improved on its RAM usage. Clustering options are also available.

关于AMQP,我想说一个非常酷的功能是“交换”,以及路由到其他交换的能力。这为您提供了更大的灵活性,并使您能够创建广泛的精心设计的路由类型,这在扩展时非常方便。关于一个好的例子,请参见:

(来源:springsource.com)

和:http://blog.springsource.org/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/

Finally, in regards to Redis, yes, it can be used as a message broker, and can do well. However, Rabbitmq has more message queuing features than Redis, as rabbitmq was built from the ground up to be a full-featured enterprise-level dedicated message queue. Redis on the other hand was primarily created to be an in-memory key-value store(though it does much more than that now; its even referred to as a swiss army knife). Still, I've read/heard many people achieving good results with Redis for smaller sized projects, but haven't heard much about it in larger applications.

下面是一个在长轮询聊天实现中使用Redis的例子:http://eflorenzano.com/blog/2011/02/16/technology-behind-convore/

其他回答

Rabbit's queues reside in memory and will therefore be much faster than implementing this in a database. A (good)dedicated message queue should also provide essential queuing related features such as throttling/flow control, and the ability to choose different routing algorithms, to name a couple(rabbit provides these and more). Depending on the size of your project, you may also want the message passing component separate from your database, so that if one component experiences heavy load, it need not hinder the other's operation.

关于你提到的问题:

polling keeping the database busy and low performing: Using Rabbitmq, producers can push updates to consumers which is far more performant than polling. Data is simply sent to the consumer when it needs to be, eliminating the need for wasteful checks. locking of the table -> again low performing: There is no table to lock :P millions of rows of task -> again polling is low performing: As mentioned above, Rabbitmq will operate faster as it resides RAM, and provides flow control. If needed, it can also use the disk to temporarily store messages if it runs out of RAM. After 2.0, Rabbit has significantly improved on its RAM usage. Clustering options are also available.

关于AMQP,我想说一个非常酷的功能是“交换”,以及路由到其他交换的能力。这为您提供了更大的灵活性,并使您能够创建广泛的精心设计的路由类型,这在扩展时非常方便。关于一个好的例子,请参见:

(来源:springsource.com)

和:http://blog.springsource.org/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/

Finally, in regards to Redis, yes, it can be used as a message broker, and can do well. However, Rabbitmq has more message queuing features than Redis, as rabbitmq was built from the ground up to be a full-featured enterprise-level dedicated message queue. Redis on the other hand was primarily created to be an in-memory key-value store(though it does much more than that now; its even referred to as a swiss army knife). Still, I've read/heard many people achieving good results with Redis for smaller sized projects, but haven't heard much about it in larger applications.

下面是一个在长轮询聊天实现中使用Redis的例子:http://eflorenzano.com/blog/2011/02/16/technology-behind-convore/

PostgreSQL 9.5

PostgreSQL 9.5合并SELECT…更新…跳过锁定。这使得实现工作队列系统变得更加简单和容易。您可能不再需要外部队列系统,因为现在很容易获取没有其他会话锁定的“n”行,并保持它们锁定,直到您提交确认工作完成为止。当需要外部协调时,它甚至可以处理两阶段事务。

外部队列系统仍然很有用,它提供了固定的功能、经过验证的性能、与其他系统的集成、水平扩展和联合的选项等等。然而,对于简单的情况,你不再需要它们了。

旧版本

你不需要这样的工具,但是使用一个可以让生活更容易。在数据库中进行排队看起来很容易,但是在实践中您会发现在关系数据库中很难正确地进行高性能、可靠的并发排队。

这就是PGQ这样的工具存在的原因。

你可以在PostgreSQL中通过使用LISTEN和NOTIFY来消除轮询,但是这并不能解决在保持高并发操作和不阻塞插入的同时,将队列顶部的条目可靠地分发给一个消费者的问题。所有您认为可以解决该问题的简单而明显的解决方案实际上在现实世界中都无法实现,而且往往会退化为效率较低的单工作者队列获取版本。

如果你不需要高并发的多工作者队列获取,那么在PostgreSQL中使用单个队列表是完全合理的。