Find answers, ask questions, and connect with our
community around the world.

Home Forums AWS AWS Certified Data Analytics – Specialty AWS Glue allows setting up triggers that react on crawler completion

  • AWS Glue allows setting up triggers that react on crawler completion

  • Konstantin Shilovskiy

    Member
    January 12, 2023 at 4:58 pm

    Here is the question

    A company has invested in upgrading its data analytics capabilities and improving data processing with its data warehouse, which includes AWS Glue, Amazon Redshift, and Amazon S3. The Data Analyst wants to modify the current ETL workflow and automate another Glue job after the AWS Glue crawler schedule completes. The manager seeks to minimize maintenance and development time.

    How can the Data Analyst achieve these requirements?

    I think the correct answer is:

    Create an AWS Glue trigger to directly start a job after the crawler completes.

    The documentation https://docs.aws.amazon.com/glue/latest/dg/about-triggers.html

    mentions this:

    A trigger that fires when a previous job or crawler or multiple jobs or crawlers satisfy a list of conditions.

    I also tried creating such a trigger in the AWS Console and it worked.

    However, this answer is marked as wrong (see attached picture).

    Could you please clarify?

  • Carlo-TutorialsDojo

    Member
    January 13, 2023 at 3:53 am

    Hello Konstantin,

    Thanks for your feedback.

    A standalone crawler configured to run on schedule cannot automatically invoke a Glue trigger to run a job. It must be started by a trigger. For example, to automatically run a job once a crawler run completes using Glue triggers, the setup should be glue trigger (scheduled) -> crawler -> glue trigger (conditional) -> Glue job.

    The option highlighted in red is suggesting the setup: glue scheduled crawler -> glue trigger (conditional) -> Glue Job.

    Let me know if this helps.

    Regards,

    Carlo @ Tutorials Dojo

Viewing 1 - 2 of 2 replies

The forum ‘AWS Certified Data Analytics – Specialty’ is closed to new discussions and replies.

Original Post
0 of 0 posts June 2018
Now