我什么时候使用SNS和SQS,为什么它们总是耦合在一起?
当前回答
来自AWS文档:
Amazon SNS允许应用程序发送时间紧迫的消息到 多个订阅者通过“推送”机制,消除了需求 定期检查或“轮询”更新。 Amazon SQS是分布式应用程序使用的消息队列服务 通过轮询模型交换消息,并可用于 分离发送和接收组件——不需要每个组件 组件要同时可用。
扇出到Amazon SQS队列
其他回答
SNS和SQS之间有一些关键的区别:
SNS supports A2A and A2P communication, while SQS supports only A2A communication. SNS is a pub/sub system, while SQS is a queuing system. You'd typically use SNS to send the same message to multiple consumers via topics. In comparison, in most scenarios, each message in an SQS queue is processed by only one consumer. With SQS, messages are delivered through a long polling (pull) mechanism, while SNS uses a push mechanism to immediately deliver messages to subscribed endpoints. SNS is typically used for applications that need real time notifications, while SQS is more suited for message processing use cases. SNS does not persist messages - it delivers them to subscribers that are present, and then deletes them. In comparison, SQS can persist messages (from 1 minute to 14 days).
单独地,Amazon SQS和SNS用于不同的用例。但是,您可以在某些场景中将它们一起使用。
来自AWS文档:
Amazon SNS允许应用程序发送时间紧迫的消息到 多个订阅者通过“推送”机制,消除了需求 定期检查或“轮询”更新。 Amazon SQS是分布式应用程序使用的消息队列服务 通过轮询模型交换消息,并可用于 分离发送和接收组件——不需要每个组件 组件要同时可用。
扇出到Amazon SQS队列
以下是两者的对比:
实体类型
队列(类似于JMS) SNS: Topic (Pub/Sub系统)
信息消费
SQS:拉取机制——消费者从SQS中轮询和拉取消息 SNS:推送机制——SNS向消费者推送消息
用例
SQS:解耦两个应用程序并允许并行异步处理 SNS: Fanout -以多种方式处理相同的消息
持久性
SQS:如果没有消费者可用,消息将被持久化一段时间(可配置的)(最多两周),因此当消息添加到队列时,消费者不必处于空闲状态。 社交网络:没有持久性。在消息到达时出现的任何一个消费者都将获得消息并删除消息。如果没有可用的消费者,则在几次重试后消息将丢失。
消费者类型
SQS:所有消费者通常都是相同的,因此以完全相同的方式处理消息(每个消息由一个消费者处理一次,尽管在极少数情况下消息可能会被重新发送) SNS:消费者可能以不同的方式处理消息
示例应用程序
SQS: Jobs framework: The Jobs are submitted to SQS and the consumers at the other end can process the jobs asynchronously. If the job frequency increases, the number of consumers can simply be increased to achieve better throughput. SNS: Image processing. If someone uploads an image to S3 then watermark that image, create a thumbnail and also send a Thank You email. In that case S3 can publish notifications to an SNS topic with three consumers listening to it. The first one watermarks the image, the second one creates a thumbnail and the third one sends a Thank You email. All of them receive the same message (image URL) and do their processing in parallel.
AWS SNS是一个发布者订阅者网络,订阅者可以在其中订阅主题,并在发布者发布该主题时接收消息。
AWS SQS是一个队列服务,它将消息存储在队列中。SQS不能传递任何消息,需要一个外部服务(lambda、EC2等)来轮询SQS并从SQS获取消息。
由于多种原因,可以将SNS和SQS结合使用。
There may be different kinds of subscribers where some need the immediate delivery of messages, where some would require the message to persist, for later usage via polling. See this link. The "Fanout Pattern." This is for the asynchronous processing of messages. When a message is published to SNS, it can distribute it to multiple SQS queues in parallel. This can be great when loading thumbnails in an application in parallel, when images are being published. See this link. Persistent storage. When a service that is going to process a message is not reliable. In a case like this, if SNS pushes a notification to a Service, and that service is unavailable, then the notification will be lost. Therefore we can use SQS as a persistent storage and then process it afterwards.
你可以看到SNS作为一个传统的主题,你可以有多个订阅者。对于一个给定的SNS主题,您可以拥有不同的订阅者,例如,包括Lambda和SQS。你也可以使用SNS发送短信甚至电子邮件。在SNS中需要考虑的一件事是,一次只能收到一条消息(通知),所以你不能利用批处理的优势。
SQS, on the other hand, is nothing but a queue, where you store messages and subscribe one consumer (yes, you can have N consumers to one SQS queue, but it would get messy very quickly and way harder to manage considering all consumers would need to read the message at least once, so one is better off with SNS combined with SQS for this use case, where SNS would push notifications to N SQS queues and every queue would have one subscriber, only) to process these messages. As of Jun 28, 2018, AWS Supports Lambda Triggers for SQS, meaning you don't have to poll for messages any more.
此外,您还可以在源SQS队列上配置DLQ,以便在发生故障时向其发送消息。在成功的情况下,消息将被自动删除(这是另一个很大的改进),因此在忘记手动删除消息的情况下,您不必担心已经处理过的消息会再次被读取。我建议看看Lambda重试行为,以更好地理解它是如何工作的。
One great benefit of using SQS is that it enables batch processing. Each batch can contain up to 10 messages, so if 100 messages arrive at once in your SQS queue, then 10 Lambda functions will spin up (considering the default auto-scaling behaviour for Lambda) and they'll process these 100 messages (keep in mind this is the happy path as in practice, a few more Lambda functions could spin up reading less than the 10 messages in the batch, but you get the idea). If you posted these same 100 messages to SNS, however, 100 Lambda functions would spin up, unnecessarily increasing costs and using up your Lambda concurrency.
但是,如果您仍然在运行传统的服务器(如EC2实例),则仍然需要轮询消息并手动管理它们。
您还有FIFO SQS队列,它保证消息的传递顺序。截至2019年11月,SQS FIFO还支持作为Lambda的事件源
尽管它们的用例有一些重叠,但SQS和SNS都有自己的焦点。
在以下情况使用社交网络:
多个订阅者是必需的 用手机发送短信或电子邮件很方便
在以下情况下使用SQS:
只需要一个订阅者 批处理很重要
推荐文章
- 警告:未受保护的私钥文件!当尝试SSH到Amazon EC2实例时
- 使用boto3连接CloudFront时,如何选择AWS配置文件
- 在亚马逊云服务器上设置FTP
- 无法将图像推送到Amazon ECR -由于“没有基本的身份验证凭据”而失败
- 如何测试AWS命令行工具的凭据
- 将Keypair添加到现有的EC2实例中
- AWS S3:您试图访问的桶必须使用指定的端点寻址
- 你会因为EC2上的“停止”实例而被收费吗?
- 下载一个已经上传的Lambda函数
- S3 - Access-Control-Allow-Origin头
- 何时使用Amazon Cloudfront或S3
- 如何处理错误与boto3?
- 什么数据存储在亚马逊EC2实例的临时存储?
- boto3 client NoRegionError:只能在某些时候指定区域错误
- AWS ssh访问“权限被拒绝(publickey)”问题