Guided Lab: Invoking Lambda Functions through Amazon SQS Queue
Description
AWS Lambda is a serverless computing service that lets you run code without provisioning or managing servers. It executes your code only when needed and scales automatically, from a few daily requests to thousands per second.
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. It allows you to send, store, and receive messages between software components at any volume without losing messages or requiring other services to be available.
Invoking Lambda functions via SQS queues is a common pattern for building scalable and resilient applications. By integrating SQS with Lambda, you can:
-
Decouple your application components.
-
Buffer and batch work for asynchronous processing.
-
Handle high-throughput workloads efficiently.
Consider an e-commerce platform where customers place orders. When an order is placed, the details are sent to an SQS queue. The queue triggers a Lambda function to process the order, update the inventory, and send confirmation emails. This decouples the order placement from order processing, improving reliability and scalability.
Prerequisites
This lab assumes you have the following:
- Basic knowledge and understanding of AWS services, Amazon SQS, and AWS Lambda
- Familiarity with Python programming language.
If you find any gaps in your knowledge, consider taking the following lab:
Objectives
By the end of this lab, you will:
- Create an Amazon SQS queue.
- Develop an AWS Lambda function that processes messages from the SQS queue.
- Configure the SQS queue as an event source for the Lambda function.
- Understand the configuration settings for integrating SQS with Lambda.
- Test the end-to-end flow by sending messages to the SQS queue and observing the Lambda function execution.
Lab Steps
Create an Amazon SQS Queue
1. Navigate to the Amazon SQS service:
- Use the search bar to find “SQS,” then select it from the list of services.
2. Create a new queue:
a. Click on “Create queue”.
b. Choose “Standard queue” for this lab.
Standard queues offer maximum throughput, best-effort ordering, and at least one delivery.
c. Enter a name for your queue (e.g., MyTestQueue).
d. Keep the default settings and click “Create Queue.”
Create an AWS Lambda Function
1. Navigate to the AWS Lambda Console
2. Create a new Lambda function using the following configurations:
- Choose Author from scratch.
- Function name: myLambdaFunction
- Select Python 3.12 as the runtime.
- Execution role:
- Select Use an Existing Role: PlayCloud-Sanboxd
- Click Create function
3. Replace the existing code with the following:
Note: We are using the old console editor for this lab. You’re welcome to use either the old or new editor, whichever you prefer; the steps remain the same, though the interface may have a slightly different appearance in the new editor.
import json
def lambda_handler(event, context):
# Print the event received from SQS
print("Received event from SQS:")
print(json.dumps(event, separators=(',', ':')))
# Process each record in the event
for record in event['Records']:
body = record['body']
print(f"Processing message: {body}")
return {
'statusCode': 200,
'body': json.dumps('Messages processed successfully!')
}
4. Click Deploy to save the changes.
Configure the SQS Queue as a Trigger for Lambda
1. Add a Trigger to the Lambda Function:
-
In the Lambda function designer, click on + Add trigger.
2. Configure the Trigger Settings:
a. Trigger configuration:
-
- Select a trigger: Choose SQS from the dropdown or search SQS in the search bar.
-
- SQS queue: Choose your queue (MyTestQueue) from the list or enter its ARN.
-
- Activate trigger: Check this box to activate the trigger immediately.
Select to activate the trigger now. Keep unchecked to create the trigger in a deactivated state for testing (recommended).
-
- Batch size (optional): Set to 2.
The number of records in each batch to send to the function. The maximum is 10,000 for standard queues and 10 for FIFO queues.
-
- Batch window (optional): Set to 3 seconds.
The maximum amount of time to gather records before invoking the function.
-
- Maximum concurrency (optional): Leave empty.
Specify a value between 2 and 1000. To deactivate, leave the box empty.
-
- Report batch item failures (optional): Leave unchecked.
Allow your function to return a partial successful response for a batch of records.
-
- Filter criteria (optional): Click Add and enter a JSON filter to process only messages that meet certain conditions.
Define any filters if needed. Example filter to process messages where order_type is express:
{
"body": {
"order_type": ["express"]
}
}
b. Click Add to attach the trigger.
c. Ensure that the Configuration Triggers of the Lambda for SQS state are Enabled.
Understanding the Configuration Settings:
- Event Source Mapping: Lambda uses an event source mapping to read items from the queue and invoke your function.
- Polling: Lambda will poll the SQS queue and process messages as they arrive.
- Batch Size: Controls how many messages are sent to the function simultaneously.
- Batch Window: The maximum time Lambda waits to fill out the batch size before invoking the function.
- Maximum Concurrency: Limits the number of concurrent function executions from this event source.
- Filter Criteria: You can only process specific messages based on their content.
Test the Setup
1. Send Messages to the SQS Queue:
- Navigate back to your SQS queue in the AWS Console.
- Click on your queue (MyTestQueue).
- Select Send and receive messages.
- In the Message body, enter a message with a specific order_type:
{
"order_id": 5678,
"customer": "Jane Smith",
"order_type": "express"
}
- Click Send message.
-
Now, send another message with a different order_type that doesn’t match the filter:
{
"order_id": 91011,
"customer": "Alice Johnson",
"order_type": "standard"
}
-
Repeat with additional messages as needed, varying the order_id, order_type, and customer field.
2. Verify Lambda Execution:
- Go back to your Lambda function.
- Click on the Monitoring tab.
- Select View logs in CloudWatch.
- In CloudWatch Logs, find the latest log stream for your function.
- Review the logs to see the messages printed by your function.
- Only the messages matching the order_type filter (e.g., “express”) should appear.
- Messages with order_type values that don’t meet the filter criteria (e.g., “standard”) will not trigger the function and, therefore, won’t appear in the logs.
Congratulations! You’ve successfully set up an AWS Lambda function triggered by an Amazon SQS queue. You learned how to configure the event source mapping, understand the various settings, and test the flow.
As a best practice, remember to delete any resources no longer in use, such as SQS queues and Lambda functions, to maintain a clutter-free AWS environment.
Thank you for joining this lab, and happy learning!