Run Scripts from GitHub

[anusha09]

Hi,

I am new to Talend. I have all bash scripts on GitHub. Is it possible to run jobs from git repository or pull the scripts from git to Talend cloud and run the jobs from there. Is this feasible? I am using tsystem component to run the bash scripts on my local machine. In Home directory path I give my local path. How do we do this if we want to fetch script from GitHub itself.

[Anonymous]

Hi,

   While you can theoretically do all the steps by doing extraction of scripts and then send them to tSystem to run bash scripts, you are reducing the capability of Talend to mere orchestrator.

    Your business logic is still stored in complex bash scripts which will be very difficlt to debug in case of any modification or bug fix. Ideally you should use the opportunity to construct the flow in proper graphical interface within Talend instead of just calling the bash scripts. It may take 1 or 2 extra days compared to calling the existing scripts to create the job.

     But over a long term perspective, it will give you return of investment since the graphical user interface will be very easy to debug and make modifications compared to Unix scripts.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[anusha09]

Thanks for responding. What you said is long term solutions. We have 100's of bash scripts sitting in Github. Is there a solution to get the script from Github directly and run in Talend?

Anusha 

[Anonymous]

Hi,

    You will have to write custom program to download the GitHub files containing zip files of your bash codes. You can embed them in Talend routines and call these routines from a tJava or somethign similar. Once the zip file is available in source location, you can unzip them and extract your bash files. You can then pass the details/commands present in these files to terminal components of Talend like tSystem.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[anusha09]

Two questions here

1.When you say download to source location, does that mean I have to download GitHub files to either my local machine or to any central repository on any server. Right? I do not want to recopy the scripts from git repository to my local directory. 

2. Can we call scripts from Github into Talend jobs directly by giving GitHub path

[JHale164]

Ok we had a similar issue so what I did was github cli and created a local copy of your repo which will create your repo folder structure locally and then you just use tsystem component to run the scripts.

You can refresh the repo using the gihub commands etc using tsystem as and when required.

Hope that helps