Guided Lab: Integrating SNS FIFO Topic with SQS FIFO Queue

Description

Amazon SNS FIFO (First-In-First-Out) topics are designed to ensure that messages are delivered in the exact order they are sent and processed only once. They guarantee strict ordering for events, which makes them perfect for use cases where maintaining the sequence of operations is critical. Unlike standard topics, SNS FIFO topics avoid duplicating messages and ensure that messages in the same group are processed sequentially, adhering to exactly-once semantics.

Amazon SQS FIFO queues are message queues designed for applications that require strict ordering and exact-once message processing. These queues ensure that messages are received and processed in the order they were sent and prevent duplicate message delivery. This behavior is essential for transactional systems, such as financial applications, where the correct sequence of operations must be preserved.

When integrated, SNS FIFO and SQS FIFO combine publish/subscribe messaging (SNS) benefits with reliable, ordered message handling (SQS). This integration allows for scalable communication between distributed systems or microservices while ensuring that messages are delivered in a strict order without duplicates.

By using both services, you can:

  • Decouple message producers from consumers: SNS FIFO allows messages to be published by different components, while SQS FIFO queues ensure the messages are processed in the exact sequence they were sent.
  • Maintain ordering guarantees: Both SNS FIFO and SQS FIFO enforce message ordering and delivery consistency.
  • Scale reliably: SNS topics can fan out messages to multiple consumers, and SQS FIFO queues ensure the sequential processing of those messages in downstream services.

In this lab, we’ll simulate an e-commerce system where customer orders must be processed in the exact sequence they are received to maintain data consistency. By integrating an SNS FIFO topic with an SQS FIFO queue, we’ll ensure that order messages are reliably and sequentially delivered to the processing system. This ensures that each order is processed correctly, avoiding issues like processing a shipping notification before payment confirmation or order creation.

Prerequisites

This lab assumes you have the following:

  • Basic knowledge and understanding of AWS services, particularly Amazon SNS and Amazon SQS

If you find any gaps in your knowledge, consider taking the following lab:

Objectives

By the end of this lab, you will:

  • Understand the key differences between SNS FIFO topics and Standard topics.
  • Create an SNS FIFO topic in AWS.
  • Create an SQS FIFO queue and subscribe it to the SNS FIFO topic.
  • Publish messages to the SNS FIFO topic and verify that they are delivered to the SQS FIFO queue in the correct order.
  • Apply appropriate permissions to allow SNS to send messages to SQS.

Lab Steps

Create a FIFO Topic

1. Navigate to the Amazon SNS service:

  • Use the search bar to find “sns,” then select it from the list of services.

2. In the left sidebar, click on Topics.

3. Click on Create topic.

4.  Follow the configurations below:

  • Choose FIFO as the topic type.
  • In the Name field, enter a name for your topic (e.g., MyFifoTopic.fifo). The .fifo suffix is mandatory for FIFO topics.
  • Enable Content-based deduplication. This feature prevents duplicate messages based on content.
  • Leave other settings as default and click Create topic.

Create a SQS FIFO Queue

1. Navigate to the Amazon SQS service:

  • Use the search bar to find “SQS,” then select it from the list of services.

2. Create a new queue:

a. Click on “Create queue”.

b. Select FIFO as the queue type.

c. Name your queue (e.g., MyFifoQueue.fifo). The .fifo suffix is required for FIFO queues.

d. In the FIFO queue settings, enable Content-based deduplication.

e. Leave other settings as default, scroll down, and click Create queue.

Subscribe the SQS FIFO Queue to the SNS FIFO Topic

1. In the SNS dashboard, navigate to the MyFifoTopic.fifo FIFO topic we created earlier. Look for the  Subscriptions section.

2. Click Create subscription.

3. Choose the Subscription Protocol:

  • In the Protocol drop-down, choose Amazon SQS.
  • In the Endpoint field, select the SQS FIFO queue you created earlier (e.g., MyFifoQueue.fifo).
  • Tick on Enable raw message delivery

  • Click Create subscription

Configure SQS Queue Permissions for SNS

1. Navigate to MyFifoQueue.fifo SQS FIFO Queue we created earlier.

2. Update the Queue Permissions:

  • Look for the  Queue policies section.

  • Click on Edit. You will be redirected to the edit page. Scroll Down.

  • Copy and paste the following policy to grant permission to the SNS FIFO topic:
{
  "Version": "2012-10-17",
  "Id": "QueuePolicy",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:<region>:<account-id>:MyFifoQueue.fifo",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:<region>:<account-id>:MyFifoTopic.fifo"
        }
      }
    }
  ]
}

Replace <region>, <account-id>, MyFifoQueue.fifo, and MyFifoTopic.fifo with your actual AWS region, account ID, and resource names.

  • Scroll down and click Save.

Publish Multiple Messages to Test FIFO Ordering

1. Publish Messages to the SNS FIFO Topic:

  • Go back to MyFifoTopic.fifo SNS FIFO Topic again.
  • Click Publish message.

2. Enter Message Details for Multiple Messages:

  • Publish three messages, ensuring each has the same Message group ID (e.g., OrderGroup1):
    • Message 1:
      • Subject: Test FIFO Message 1
      • Message body: This is the first message in the FIFO group.

    • Message 2:
      • Subject: Test FIFO Message 2
      • Message body: This is the second message in the FIFO group.

    • Message 3:
      • Subject: Test FIFO Message 3
      • Message body: This is the third message in the FIFO group.

  • Ensure Message Group ID: All three messages should have the Message group ID set to OrderGroup1, ensuring that they are processed in order.

Verify FIFO Behavior in the SQS Queue

1. Poll the SQS Queue for Messages:

  • Navigate to the SQS Console, and select MyFifoQueue.fifo.
  • Click Send and receive messages. Notice there are three messages available. This means the Publish message from the SNS was received successfully by the SQS Queue.

  • Click Poll for messages to retrieve messages sent to the queue.
  • Click on each ID. The messages should appear in the exact order they were sent:
    • Message 1: This is the first message in the FIFO group.
    • Message 2: This is the second message in the FIFO group.
    • Message 3: This is the third message in the FIFO group.

Congratulations! You successfully created an SNS FIFO topic and subscribed to an Amazon SQS FIFO queue. You published multiple messages to the SNS FIFO topic and verified that the SQS FIFO queue received them in the correct order. You also applied the IAM policy to allow SNS to send messages to SQS. This setup demonstrates the power of FIFO queues and topics to ensure ordered and reliable message delivery for sensitive applications such as financial transactions, inventory management, etc.

As a best practice, remember to delete any resources no longer in use, such as SQS queues and SNS topics, to maintain a clutter-free AWS environment.

Thank you for joining this lab, and happy learning!

Skip to content