topic Can we write a custom PySpark code Talend? in Talend Studio

Can we write a custom PySpark code Talend?

Anonymous — Sat, 16 Nov 2024 07:44:37 GMT

Hi,
Is there any option in Talend bigdata commercial edition to write a custom PySpark (spark code in python) code?

Re: Can we write a custom PySpark code Talend?

Anonymous — Thu, 30 Aug 2018 19:45:58 GMT

Hi,

You cannot use PySpark code in Talend Spark jobs directly. Please refer the below help document which specifies how Talend Spark job is working.

https://help.talend.com/reader/bYQYL576UebCj3V76mUoeA/svekiZRvdXnsuccbPLg6Ow

But the advantage of Talend is that you can integrate other applications by writing the right interface custom routines by yourself. In your case, it will be slightly circuitous way as the same objectives can be achieved using Talend java custom routines.

If the answer has helped to resolve your query, could you please mark the topic as closed?

Warm Regards,

Nikhil Thampi

Re: Can we write a custom PySpark code Talend?

Anonymous — Thu, 30 Aug 2018 19:50:45 GMT

Hi Nikhil,

Thanks for your reply. Can you share some help document or few examples on how to use java custom routines.

Re: Can we write a custom PySpark code Talend?

Anonymous — Thu, 30 Aug 2018 20:00:38 GMT

Hi Rajesh,

Please refer the below help document and KB articles for your reference.

https://help.talend.com/reader/BnnM0hh3643D9Vq15udPtA/KFMROs1xk5w9fefl7uf1AA

https://community.talend.com/t5/Design-and-Development/Create-a-user-routine-and-call-it-in-a-Job/ta-p/21665

If the reply has helped to answer your query, could you please mark the topic as closed? It will help to enrich the Talend Community. Kudos are also welcome 🙂

Warm regards,

Nikhil Thampi