我有一个用例,将有数据流到来,我不能以相同的速度消费它,需要一个缓冲区。这可以使用SNS-SQS队列来解决。我后来才知道,Kinesis解决了同样的目的,所以有什么不同?为什么我应该喜欢(或不应该喜欢)运动?
当前回答
定价模型是不同的,因此根据您的用例,一种或另一种可能更便宜。使用最简单的情况(不包括SNS):
SQS对每个消息收费(每个64 KB算作一个请求)。 Kinesis对每个分片每小时收费(一个分片最多可以处理1000条消息或1 MB/秒),也对您放入的数据量收费(每25 KB)。
考虑到当前的价格,不考虑免费级别,如果您以最大消息大小每天发送1gb的消息,Kinesis的成本将远远高于SQS (Kinesis每月10.82美元,SQS每月0.20美元)。但如果你每天发送1tb, Kinesis会稍微便宜一些($158/月vs. SQS $201/月)。
详细信息:SQS每百万请求收费0.40美元(每次64 KB),因此每GB收费0.00655美元。以每天1gb计算,每月不到0.20美元;按每天1tb计算,每月的费用略高于201美元。
Kinesis charges $0.014 per million requests (25 KB each), so $0.00059 per GB. At 1 GB per day, this is less than $0.02 per month; at 1 TB per day, it is about $18 per month. However, Kinesis also charges $0.015 per shard-hour. You need at least 1 shard per 1 MB per second. At 1 GB per day, 1 shard will be plenty, so that will add another $0.36 per day, for a total cost of $10.82 per month. At 1 TB per day, you will need at least 13 shards, which adds another $4.68 per day, for a total cost of $158 per month.
其他回答
请记住,这个答案在2015年6月是正确的
在研究了这个问题一段时间后,我心里有同样的问题,我发现SQS(带SNS)是大多数用例的首选,除非消息的顺序对您很重要(SQS不保证消息的FIFO)。
Kinesis有2个主要优势:
您可以从多个应用程序读取相同的消息 如果需要的话,您可以重新阅读邮件。
这两个优点都可以通过使用SNS作为SQS的扇出来实现。这意味着消息的生产者只向SNS发送一条消息,然后SNS将消息分散到多个SQSs,每个使用者应用程序一个SQSs。通过这种方式,您可以拥有尽可能多的消费者,而无需考虑分片容量。
此外,我们还增加了一个订阅SNS的SQS,可以保存14天的消息。在正常情况下,没有人从这个SQS中读取数据,但如果出现让我们想要倒带数据的错误,我们可以轻松地从这个SQS中读取所有消息,并将它们重新发送到SNS。而Kinesis仅提供7天的留存。
总之,SNS+SQSs更简单,提供了大部分功能。在我看来,你需要一个非常有力的案例来选择Kinesis。
另一件事:Kinesis可以触发Lambda,而SQS不能。因此,对于SQS,您要么必须提供一个EC2实例来处理SQS消息(并在失败时处理它),要么必须有一个预定的Lambda(它不能扩大或缩小—每分钟只能得到一个)。
编辑:这个答案不再正确。自2018年6月起,SQS可以直接触发Lambda
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
定价模型是不同的,因此根据您的用例,一种或另一种可能更便宜。使用最简单的情况(不包括SNS):
SQS对每个消息收费(每个64 KB算作一个请求)。 Kinesis对每个分片每小时收费(一个分片最多可以处理1000条消息或1 MB/秒),也对您放入的数据量收费(每25 KB)。
考虑到当前的价格,不考虑免费级别,如果您以最大消息大小每天发送1gb的消息,Kinesis的成本将远远高于SQS (Kinesis每月10.82美元,SQS每月0.20美元)。但如果你每天发送1tb, Kinesis会稍微便宜一些($158/月vs. SQS $201/月)。
详细信息:SQS每百万请求收费0.40美元(每次64 KB),因此每GB收费0.00655美元。以每天1gb计算,每月不到0.20美元;按每天1tb计算,每月的费用略高于201美元。
Kinesis charges $0.014 per million requests (25 KB each), so $0.00059 per GB. At 1 GB per day, this is less than $0.02 per month; at 1 TB per day, it is about $18 per month. However, Kinesis also charges $0.015 per shard-hour. You need at least 1 shard per 1 MB per second. At 1 GB per day, 1 shard will be plenty, so that will add another $0.36 per day, for a total cost of $10.82 per month. At 1 TB per day, you will need at least 13 shards, which adds another $4.68 per day, for a total cost of $158 per month.
摘自AWS文档:
We recommend Amazon Kinesis Streams for use cases with requirements that are similar to the following: Routing related records to the same record processor (as in streaming MapReduce). For example, counting and aggregation are simpler when all records for a given key are routed to the same record processor. Ordering of records. For example, you want to transfer log data from the application host to the processing/archival host while maintaining the order of log statements. Ability for multiple applications to consume the same stream concurrently. For example, you have one application that updates a real-time dashboard and another that archives data to Amazon Redshift. You want both applications to consume data from the same stream concurrently and independently. Ability to consume records in the same order a few hours later. For example, you have a billing application and an audit application that runs a few hours behind the billing application. Because Amazon Kinesis Streams stores data for up to 7 days, you can run the audit application up to 7 days behind the billing application. We recommend Amazon SQS for use cases with requirements that are similar to the following: Messaging semantics (such as message-level ack/fail) and visibility timeout. For example, you have a queue of work items and want to track the successful completion of each item independently. Amazon SQS tracks the ack/fail, so the application does not have to maintain a persistent checkpoint/cursor. Amazon SQS will delete acked messages and redeliver failed messages after a configured visibility timeout. Individual message delay. For example, you have a job queue and need to schedule individual jobs with a delay. With Amazon SQS, you can configure individual messages to have a delay of up to 15 minutes. Dynamically increasing concurrency/throughput at read time. For example, you have a work queue and want to add more readers until the backlog is cleared. With Amazon Kinesis Streams, you can scale up to a sufficient number of shards (note, however, that you'll need to provision enough shards ahead of time). Leveraging Amazon SQS’s ability to scale transparently. For example, you buffer requests and the load changes as a result of occasional load spikes or the natural growth of your business. Because each buffered request can be processed independently, Amazon SQS can scale transparently to handle the load without any provisioning instructions from you.
这些技术的语义不同,因为它们是为支持不同的场景而设计的:
SNS/SQS:流中的项之间没有关联 运动:流中的项目相互关联
让我们通过例子来理解其中的区别。
Suppose we have a stream of orders, for each order we need to reserve some stock and schedule a delivery. Once this is complete, we can safely remove the item from the stream and start processing the next order. We are fully done with the previous order before we start the next one. Again, we have the same stream of orders, but now our goal is to group orders by destinations. Once we have, say, 10 orders to the same place, we want to deliver them together (delivery optimization). Now the story is different: when we get a new item from the stream, we cannot finish processing it; rather we "wait" for more items to come in order to meet our goal. Moreover, if the processor process crashes, we must "restore" the state (so no order will be lost).
一旦一个项目的处理不能与另一个项目的处理分离,我们就必须有运动语义,以便安全地处理所有的情况。
推荐文章
- 为什么我应该使用亚马逊Kinesis而不是SNS-SQS?
- 如何重命名AWS S3 Bucket
- AWS ECS中的任务和服务之间有什么区别?
- 亚马逊SimpleDB vs亚马逊DynamoDB
- 亚马逊ECS和亚马逊EC2有什么区别?
- 我如何知道我在S3桶中存储了多少对象?
- S3 Bucket操作不应用于任何资源
- 将AWS凭证传递给Docker容器的最佳方法是什么?
- 当权限为S3时,AccessDenied for ListObjects for S3 bucket:*
- 电子邮件地址未验证(AWS SES)
- 使用Boto3将S3对象作为字符串打开
- AWS VPC - Internet网关vs. NAT
- 如何在AWS Lambda中加载npm模块?
- 亚马逊S3 -如何修复“我们计算的请求签名与签名不匹配”错误?
- 警告:未受保护的私钥文件!当尝试SSH到Amazon EC2实例时