Amazon SNS Now Has FIFO
Here’s How It Works
About a week ago AWS introduced FIFO topics for Amazon SNS. FIFO helps preserve message order and ensures there is no duplicate message delivery to SNS subscribers.
Amazon Simple Notification Service is used as a pub/sub messaging system, where a message is published to a topic and delivered to all subscribers of that topic. SNS allows message delivery to a large number of subscribers, who can each set filter policies to only receive messages relevant to them.
You can use topics to fan-out messages to multiple queues or applications to help decouple microservices and create distributed systems or serverless applications.
With SNS FIFO, message ordering and deduplication can be achieved.
Previously, developers had to add a message ID manually in order to help preserve the order of messages published to the topic. Now that there is SNS FIFO, you can configure a message group by including message group IDs when publishing a message to a FIFO topic. FOr each message group ID, all messages are sent and delivered in order of their arrival. This helps you ensure the delivery of messages are in the correct order that they are published in. Messages published to SNS FIFO is passed to any subscribed FIFO SQS queue as well.
Distributed systems and applications sometimes generate duplicate messages, which can cause problems if a message is processed twice. You can avoid duplicated message deliveries from the topic with FIFO by enabling content-based deduplication or by adding a deduplication ID to the messages published. With content-based deduplication, SNS uses an SHA-256 hash to generate a message deduplication ID using the body of the message. Any duplicated messages sent within a specified time interval is accepted but not delivered. If an SQS FIFO queue is subscribed to an SNS FIFO topic, the deduplication ID is passed to the queue and SQS uses it to avoid duplicate messages being sent as well.
One way to simplify your application implementation is to use SNS FIFO and SQS FIFO together. A common scenario where you use FIFO topics and queues is to receive updates that need to be processed in the order they need to be. One example is an application where when customers create an order, the order ID is stored in DynamoDB and also stored into Amazon S3.
To try out the new SNS FIFO functionality, I have very quickly created a mock application with basic functionality to demonstrate the message deduplication and message ordering of SNS FIFO topics.
All Resources for Demo:
- 1 SNS FIFO Topic
- 2 SQS FIFO Topics
- AWS CLI
jqcommand-line utility
Creating the SQS queues and SNS topics
The first step is to go and create an SNS FIFO topic Order.fifo with content-based message deduplication, with other settings left on default:
I then create two SQS FIFO queues dynamodb.fifo and s3.fifo and leave it with default settings:
I then subscribe the two SQS queues to the SNS topic:
An access policy for both queues has to be created to grant the SNS topic permissions to send messages to the queues. This will allow each queue to receive the messages published by the Order.fifo topic. The two access policies are shown below:
Note: If you want to copy-paste this access policy, take note to update the resource ARNs to fit your queues and topics.
I will now send three messages to the topic using the same message group ID:
This is Order 1This is Order 2This is Order 1
After publishing, you can see that only 2 messages were received even though I published 3 messages:
This shows that the FIFO topic’s content-based message deduplication works! The 3 messages sent contained a duplicate message, which was sent but not received to the subscribed queues.
To show you the actual messages received in the queue, I will use AWS CLI to receive the messages from the SQS queues. If you have not installed and configured AWS CLI, you can refer to my guide here.
I use the aws sqs receive-message command as well as the jq command-line utility to format the output and receive the message body. Here are the messages in each queue:
$ aws sqs receive-message --queue-url https://sqs.ap-southeast-1.amazonaws.com/462574577754/dynamodb.fifo | jq '.Messages[].Body | fromjson | .Message'"This is Order 1"
"This is Order 2"$ aws sqs receive-message --queue-url https://sqs.ap-southeast-1.amazonaws.com/462574577754/s3.fifo | jq '.Messages[].Body | fromjson | .Message'"This is Order 1"
"This is Order 2"
As you can see, only the 2 messages with the unique message body content were delivered to the queues in the order they were sent.
And that’s on SNS FIFO topics! Feel free to leave some comments on what you are going to use this new feature for, I would love to hear your ideas and projects you are working on too!
