topic Re: Error to import module in python using TOS in Talend Studio

Error to import module in python using TOS

WSyahirah21 — Fri, 15 Nov 2024 23:17:01 GMT

Im currently trying to run a spark job where the job will use a module that is imported from a python script. This job works when I run it in terminal as I run the spark job in the same directory where the python script (for library) is stored.

However, when I tried to run the spark job in TOS, it returns this error:

from filename import full_ingestion

ImportError: No module named filename

I assumed that the talend cant found the library, so I used tLibraryLoad to import the new library to TOS and set it up as below:

In tLibraryLoad_1 :

basic setting : filename.py (this is the library script I wrote in python)

advanced setting : import full_ingestion (full_ingestion is a function in filename.py)

In tSystem component:

-> In spark job script , put import full_ingestion

However, I received this error.

Am i do a correct way to import a library created in python script to the spark job in TOS.

Re: Error to import module in python using TOS

gjeremy1617088143 — Mon, 24 Jan 2022 08:02:58 GMT

HI, i think tLibraryLoad is for java library, if you want to execute pyton script you surely have to use tSystem component and use a cmd to run the script in the tSystemComponent. Send me love and kudos

Re: Error to import module in python using TOS

WSyahirah21 — Mon, 24 Jan 2022 08:20:30 GMT

Do u know how is it possible for me to use python module and import it into a script? Im currently using tSystemComponent to run the python script but it requires me to import the python module first.

exp:

from python_filename import anylibrary

Re: Error to import module in python using TOS

gjeremy1617088143 — Mon, 24 Jan 2022 08:45:20 GMT

you have to reproduce the command you use when you say 'This job works when I run it in terminal as I run the spark job in the same directory where the python script (for library) is stored'.

Re: Error to import module in python using TOS

gjeremy1617088143 — Mon, 24 Jan 2022 08:47:53 GMT

I also found this workaround maybe it can help you :

https://stackoverflow.com/questions/42344399/error-executing-python-script-file-through-talend-tsystem

maybe you don't have to install it in this specific folder but just in the plugin folder

Re: Error to import module in python using TOS

WSyahirah21 — Mon, 24 Jan 2022 09:37:05 GMT

#1 The one i run in terminal succeed is because i put the python module script using the same dir. however, in Talend studio i think its using the diff dir as in terminal, where the dir in TOS is:

/data/Talend-7.3.1/jobserver/agent/TalendJobServersFiles/repository/filepath_20220124_173021_mM8rA/jobname/subfilepath/jobname_0_1/resources/. so Im not so sure how should i run the python script for module or how should I pass the module to the spark job script

#2 that solution doesnt work as it can simply install using pip install unidecode. However, mine is using python script.

Re: Error to import module in python using TOS

ddduser1643959971 — Sat, 12 Feb 2022 18:59:17 GMT

I have a csv File

name; FirstName ; numero;adresse

1 a;b;12,xx

2 c;b;13;yy

3 x;y;47;zz

4 e;r;45;tt

I want identify the doublon row by FirstName( here row number 1 and 2 firstName =b )

and i want to update the csv file to add new column "status" NotUnique like this

name; FirstName ; numero;adresse;status

1 a;b;12,xx;NotUnique

2 c;b;13;yy;NotUnique

3 x;y;47;zz;unique

4 e;r;45;tt;unique

can some one help me

Thank you very mutch in advance

Re: Error to import module in python using TOS

NNekkanti1658462243 — Wed, 31 Aug 2022 09:51:45 GMT

Dear friends,

I am trying to run python script in TAC,

I am getting this error.Please help me out .It is very critical for me.

Traceback (most recent call last):

File "C:\Users\Talenderuser1\Desktop\Talend\python\LuLu_Client_machine.py", line 1, in <module>

from bs4 import BeautifulSoup

ModuleNotFoundError: No module named 'bs4'

Your responce is very valuable to me.