Talend Job execution using Apache Airflow

TalendSolutionExpert — Thu, 24 Oct 2024 11:14:05 GMT

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. Airflow uses Directed Acyclic Graph (DAG) to create workflows or tasks. For more information, see the Apache Airflow Documentation page.

This article shows you how to leverage Apache Airflow to orchestrate, schedule, and execute Talend Data Integration (DI) Jobs.

Environment

Apache Airflow 1.10.2
Nexus 3.9
WinSCP 5.15
PuTTY

Prerequisites

Apache Airflow installed on a server (follow the Installing Apache Airflow on Ubuntu/AWS installation instructions).
Python 2.7 installed on the Airflow server.
Java 1.8 installed on the Airflow server.
Access to the Nexus server from the Airflow server (in this example, both Nexus and Airflow are installed on the same server).
Talend 7.x Jobs published to the Nexus repository. (For more information on how to set up a CI/CD pipeline to publish Talend Jobs to Nexus, see Configuring Jenkins to build and deploy project items in the Talend Help Center.)
Access to the setup_files.zip file (attached to this article).

Process flow

Develop Talend DI Jobs using Talend Studio.
Publish the DI Jobs to the Nexus repository using Talend CI/CD module.
Execute the Directed Acyclic Graph (DAG) in Apache Airflow:
- The first step in DAG is to download the Job executable from Nexus using the customized script.
- The second step is to execute the downloaded Job.

Configuration and execution

Login to the Airflow server through SSH using WinSCP or PuTTY.
Create two folders named jobs and scripts under the AIRFLOW_HOME folder.
Extract the setup_files.zip, then copy the shell scripts (download_job.sh and delete_job.sh) to the scripts folder.
Copy the talend_job_dag_template.py file from the setup_files.zip to your local machine and update the following:
- nexus_host
- nexus_port
- airflow_home
- nexus_repo
- job_group_id
- job_name
- job_version
Also, update the default_args dictionary based on your requirements.

For more information, see the Apache Airflow documentation: Default Arguments.
The DAG template provided is programmed to trigger the task externally. If you plan to schedule the task, update the schedule_interval parameter under the DAG for airflow task with values based on your scheduling requirements.

For more information on values, see the Apache Airflow documentation: DAG Runs.
Rename the updated template file and place it in the dags folder under the AIRFLOW_HOME folder.
After the Airflow scheduler picks up the DAG file, a compiled file with the same name and with a .pyc extension is created.
Refresh the Airflow UI screen to see the DAG.

Note: If the DAG is not visible on the User Interface under the DAGs tab, restart the Airflow webserver and the Airflow scheduler.
To schedule the task, toggle the button to On. You can also run the task manually.
Monitor the run status on the Airflow UI.

Conclusion

In this article, you learned how to author, schedule, and monitor workflows from the Airflow UI, and how to download and trigger Talend Jobs for execution.

Re: Talend Job execution using Apache Airflow

jsnasli — Thu, 24 Oct 2024 08:36:47 GMT

Hi,

Where can I find the setup_files.zip file.

I did not found the attached file to this article.

Thank you

Re: Talend Job execution using Apache Airflow

Sonja_Bauernfeind — Thu, 24 Oct 2024 11:21:23 GMT

Hello @jsnasli

Please check now!

All the best,
Sonja

article Talend Job execution using Apache Airflow in Official Support Articles

Talend Job execution using Apache Airflow

Environment

Prerequisites

Process flow

Configuration and execution

Conclusion

Re: Talend Job execution using Apache Airflow

Re: Talend Job execution using Apache Airflow