topic Is it possible to import python modules in Talend. in Talend Studio

Is it possible to import python modules in Talend.

WSyahirah21 — Fri, 15 Nov 2024 23:16:35 GMT

I'm running the spark job where it needs to use the library from the python module I created. The snippet of spark script:

from loaders import ingestion

logging.basicConfig(level=logging.INFO)

log = logging.getLogger('jdbc-loader')

However, once I run the spark job, it returns the following error.

File "/data/Talend-7.3.1/jobserver/agent/TalendJobServersFiles/repository/<path>", line 17, in <module>

from loaders import ingestion

ImportError: No module named loaders

Is there any ways to pass the

python module into a script in Talend Studio?

Re: Is it possible to import python modules in Talend.

Anonymous — Tue, 25 Jan 2022 07:18:29 GMT

Hello,

Are you able to run your python script fine from CMD window? There is something lack of python module in your python script?

So far, there are two ways to import and use an external jar in talend

- Use tLibraryLoad to load the jar in the job.

- Create a custom routine, import the jar in the routine and then call the routine in the job.

It maybe not applicable for python module.

Best regards

Sabrina

Re: Is it possible to import python modules in Talend.

WSyahirah21 — Tue, 05 Apr 2022 06:36:31 GMT

Hi Sabrina, it has been a while, but is it possible to use the python module if I add the resource into PYTHONPATH environment? I wanted to try it out.

However, to try that way, I need to put the python module into the directory of talend resources in server.

My question is, where can I locate the directory of the following resource in terminal.

So far, I only can see the directory of resources per job execution. E.g.

/data/Talend-7.3.1/jobserver/agent/TalendJobServersFiles/repository/INGESTION_20220402_130416_WoC1C/Ingestion_Dev/ingestion/ingestion_dev_0_13/resources/spark_jdbc_0.1.py

Re: Is it possible to import python modules in Talend.

Anonymous — Wed, 06 Apr 2022 02:15:58 GMT

Hello,

I'm not sure that it is possible to use the python module if you add the resource into PYTHONPATH environment and I need to check and make an investigation on it with our manager. I will come back to you as soon as possible.

Best regards

Sabrina

Re: Is it possible to import python modules in Talend.

WSyahirah21 — Tue, 12 Apr 2022 19:58:09 GMT

Hi xdshi,

Issue solved. In order to use the python module in talend, I did following steps:

Added modulefile.zip.tgz into resources, unarchived that file using tFileUnarchive.
Set the java.nio.file.Paths.get(context.pylibs).getParent().toString() in Extraction Directory parameter to read the parent file of modulefile.
Use the python module in pyspark script as usual using import statement.

Thanks for you respond!

Re: Is it possible to import python modules in Talend.

Anonymous — Wed, 13 Apr 2022 02:51:36 GMT

Hello,

Thanks for letting us know you have resolved this issue by yourself and sharing solution with us.

Best regards

Sabrina