Navigating job dependencies in AWS glue – Managing ETL workflows

AWS Glue manages dependencies between jobs using triggers. Triggers can start jobs based on the completion status of other jobs, making it possible to create ETL workflows where one job’s output is another job’s input.

Types of triggers:

Scheduled Triggers: Start jobs at specified times.

On-Demand Triggers: Start jobs manually.

Job Completion Triggers: Start jobs based on the completion status of other jobs.

Job completion triggers

To manage dependencies between jobs, Job Completion Triggers are especially important. They allow you to start jobs when the triggering jobs succeed, fail, or stop, and you can use them to set up complex job workflows with multiple dependencies.

Creating job dependencies

Scenario:

You have three jobs: JobA, JobB, and JobC.

JobB should run after the successful completion of JobA.

JobC should run after the successful completion of JobB.

Steps:

Create jobs in AWS Glue

Navigate to AWS Glue Console.

Create the three jobs, JobA, JobB, and JobC.

Create triggers

Create a trigger TriggerAB to start JobB when JobA succeeds.

Create another trigger TriggerBC to start JobC when JobB succeeds.

Python/Boto3 example

Using AWS SDK for Python (Boto3), you can create jobs and triggers as follows:

import boto3
glue = boto3.client('glue')
# Define Job Names
job_a = 'JobA'
job_b = 'JobB'
job_c = 'JobC'
# Create Jobs (Assume that the job scripts and other parameters are already defined)
glue.create_job(Name=job_a, /* other parameters */)
glue.create_job(Name=job_b, /* other parameters */)
glue.create_job(Name=job_c, /* other parameters */)
# Create Triggers
trigger_ab = {
    'Name': 'TriggerAB',
    'Type': 'CONDITIONAL',
    'Actions': [{'JobName': job_b, 'Arguments': {}}],
    'Predicate': {
        'Conditions': [
            {'LogicalOperator': 'EQUALS', 'JobName': job_a, 'State': 'SUCCEEDED'}
        ]
    }
}
glue.create_trigger(**trigger_ab)
trigger_bc = {
    'Name': 'TriggerBC',
    'Type': 'CONDITIONAL',
    'Actions': [{'JobName': job_c, 'Arguments': {}}],
    'Predicate': {
        'Conditions': [
            {'LogicalOperator': 'EQUALS', 'JobName': job_b, 'State': 'SUCCEEDED'}
        ]
    }
}
glue.create_trigger(**trigger_bc)