CloudWatch Event ECS question is flawed

[SamStephens]

I’ve just taken Timed Mode Set 1 – AWS Certified Solutions Architect Associate, and one of the answers is not correct.

The question is

49. Question

An application is hosted in an AWS Fargate cluster that runs a batch job whenever an object is loaded on an Amazon S3 bucket. The minimum number of ECS Tasks is initially set to 1 to save on costs, and it will only increase the task count based on the new objects uploaded on the S3 bucket. Once processing is done, the bucket becomes empty and the ECS Task count should be back to 1.

Which is the most suitable option to implement with the LEAST amount of effort?

The correct answer according to the test is

Set up a CloudWatch Event rule to detect S3 object PUT operations and set the target to the ECS cluster with the increased number of tasks. Create another rule to detect S3 DELETE operations and set the target to the ECS Cluster with 1 as the Task count.

This answer does not actually meet the stated requirements. The requirement is to “increase the task count based on the new objects uploaded on the S3 bucket”. This requirements reads as saying each uploaded object should increase the task count by one, so at any time the ECS task count is equal to the number of objects in the S3 bucket plus one. However, it’s not possible to dynamically set the task count to match the number of objects uploaded to S3, as Cloudwatch Event Targets can only set the ECS Task Count to a predefined static number.

You could reword the requirement to say something like “The minimum number of ECS Tasks is initially set to 1 to save on costs, and it will set a higher task count when new objects are uploaded to the S3 bucket”.

Note however even with that rewording of the question, the proposed solution isn’t good architecturally either. The problem is the DELETE happens once the first object is processed. Imagine the scenario:

• 500 S3 objects are uploaded to the bucket.
• The Cloudwatch Event Rule sets the ECS Task count to an increased number. That number isn’t specified in the question, but lets say that number is 10 for this example.
• 10 ECS Jobs will process 10 S3 Objects.
• The first job completes and deletes its associated S3 Object.
• The ECS task count is set back to 1.

Now the ECS cluster has a task capacity of one to process the remaining 490 S3 objects; we don’t have parallelism any more because the capacity has been reduced too early.

I’d argue that the Lamba answer is actually the correct answer, because Lambda is needed to actually manage ECS capacity efficiently based on how many objects remain to be processed; it’s not possible to solve this effectively by setting the ECS Task Count directly from the Cloudwatch Event Rule.

• This discussion was modified 3 years ago by  SamStephens. Reason: Better formatting

[SamStephens]

• This reply was modified 3 years ago by  SamStephens. Reason: Removing my test reply