Guided Lab: Automated File Processing with S3 Event Notifications and Lambda function
Description
You can turn on S3 event notifications to send notifications on S3-related events, such as object uploads or deletions. These notifications can be configured to target AWS services like Lambda functions, SQS queues, or SNS topics.
In this lab, you’ll set up an S3 bucket to generate an event notification, which in turn will trigger a Lambda function to process the uploaded file. This combination of S3 events and Lambda functions is a common pattern in serverless architectures.
Prerequisites
This lab assumes you have some prior experience with the basic creation of S3 buckets and Lambda functions.
If you find any gaps in your knowledge, consider going through the following labs:
- Creating an Amazon S3 bucket
- Creating an AWS Lambda function
Objectives
Imagine you’re a solutions architect at a financial company. Your job is to provide the audit team with the transaction records they need for their checks. But there’s a catch due to privacy rules: they can’t see full credit card numbers. So, your challenge is to make sure they only get to see the last four digits of each card number. This way, they have just enough information to do their job without compromising customer data.
In this lab, you’ll set up this system. You’ll create an S3 bucket, configure its event notifications to trigger a Lambda function, and write the function logic to process and store a redacted version of the uploaded file in a designated folder within the same bucket.
In this lab, you will:
- Learn how to enable S3 event notifications to invoke a Lambda function
- Learn how to read from and upload files to an S3 bucket using the AWS SDK for Python (Boto3)
Subscribe to access AWS PlayCloud LabsLab Steps
Creating the Lambda function
1. Create a Lambda function called redact-transaction-record and choose Python as the runtime.
2. For the execution role, click the Use an existing role option, then choose PlayCloud-Sandbox.
3. Click the Create function.
Deploying the Lambda function code
1. Download the following ZIP file on your local machine.
https://media.tutorialsdojo.com/public/redact-transaction-record.zip
2. On the AWS Lambda console, click the Upload from dropdown box, then click .zip file.
3. A dialog box will appear. Upload the ZIP file you downloaded and click on Save.
Configuring S3 Event notification
1. Create an S3 bucket with the name transactions-<your-name> (for example, transactions-carlo) in the N. Virginia region. S3 bucket names need to be globally unique. Include your name or any arbitrary characters to ensure that your bucket name won’t conflict with existing ones.
2. Once created, click the Properties section of your bucket.
3. Scroll down the Event notifications setting and click Create event notification.
4. In the General configuration section, enter the following details:
5. For Event Types, tick the Put checkbox.
6. For Destination, choose the redact-transaction-record Lambda function. Then, click on Save changes.
Testing the Configuration
Now that we’re done setting up all the required resources, let’s do a quick test.
When we created the event notification, note that we have configured both a Prefix and a Suffix. In this setup, ‘input/’ is used as the Prefix, which means the event will only trigger for files uploaded to the ‘input’ folder within the bucket. The Suffix ‘.csv’ ensures that the event is triggered only for files with a ‘.csv’ extension. We use these filters to prevent unnecessary S3 notifications to our Lambda functions, ensuring they run only in response to relevant files.
To start the test:
1. Create a folder named ‘input’ in the S3 bucket.
2. Download the transactions.csv file into your computer.
3. Open up the transactions.csv. You’ll notice that the file contains 5 transactions, with the card numbers fully displayed.
4. Upload the transactions.csv to the input folder.:
5. Go back to the Objects section of your S3 bucket. You’ll find a new output folder created in the bucket.
6. Click the output folder.
7. You should see a new file called redacted_transactions.csv. Download it to your local folder and open it. This file contains the same transaction records, but with the first 12 digits of each credit card number redacted.