Convert Pyspark ETL job to Talend ETL job

[Anonymous]

Hi,

Is it feasible to convert custom Pyspark jobs to Talend native jobs?

I have a  requirement to integrate existing Pyspark jobs with Talend.  Is there a way I can trigger Pyspark jobs from Talend? If yes , then what should be the approach? 

[Anonymous]

Do you mean Spark jobs you design using talend?

Sorry, I don't understand

[Anonymous]

Hi,

    There are no Talend Studio components which can directly invoke PySpark. But the other way to do the process is by using tSSH or tSystem components, where you can invoke the commands through Shell mode.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[ShikhaSharma]

This is regarding migration of pyspark jobs to Talend. In other words, can we call pyspark jobs through talend.

We have a requirement where we have 300 pyspark jobs and we want to call them through talend.

[Anonymous]

@asharma 

There are no direct components in Talend to call PySpark. But once your provide the necessary execution privileges, you should be able to call the PySpark through command line.

https://spark.apache.org/docs/0.9.0/python-programming-guide.html

 This feature can be completed through the terminal components of Talend like tSSH.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂