AWS Glue allows setting up triggers that react on crawler completion

[Konstantin Shilovskiy]

Here is the question

A company has invested in upgrading its data analytics capabilities and improving data processing with its data warehouse, which includes AWS Glue, Amazon Redshift, and Amazon S3. The Data Analyst wants to modify the current ETL workflow and automate another Glue job after the AWS Glue crawler schedule completes. The manager seeks to minimize maintenance and development time.

How can the Data Analyst achieve these requirements?

I think the correct answer is:

Create an AWS Glue trigger to directly start a job after the crawler completes.

The documentation https://docs.aws.amazon.com/glue/latest/dg/about-triggers.html

mentions this:

A trigger that fires when a previous job or crawler or multiple jobs or crawlers satisfy a list of conditions.

I also tried creating such a trigger in the AWS Console and it worked.

However, this answer is marked as wrong (see attached picture).

Could you please clarify?

[Carlo-TutorialsDojo]

Hello Konstantin,

Thanks for your feedback.

A standalone crawler configured to run on schedule cannot automatically invoke a Glue trigger to run a job. It must be started by a trigger. For example, to automatically run a job once a crawler run completes using Glue triggers, the setup should be glue trigger (scheduled) -> crawler -> glue trigger (conditional) -> Glue job.

The option highlighted in red is suggesting the setup: glue scheduled crawler -> glue trigger (conditional) -> Glue Job.

Let me know if this helps.

Regards,

Carlo @ Tutorials Dojo