topic Not able to iterate files in Talend Spark using tLoop in Talend Studio

Not able to iterate files in Talend Spark using tLoop

pragna_k — Sat, 16 Nov 2024 07:42:01 GMT

Is there a way to iterate files in Spark ? Like tfilelist . I know we can give multiple files as input in tfiledelimited input by giving wild card char "*". But my requirement is to iterate each xml file and create corresponding output file with same name. May i know if u have any suggestions on how to achieve this in spark ?

Here is the solution (Screenshot below) that i tried with tLoop which also not working.

In the below code, context.FileList= concatenated list of files with ",", eg: file1,file2,file3 and so on...

tJava component Code ==>

List<String> FileList = Arrays.asList(context.FileList.split(","));
context.FileListN = context.FileList;

System.out.println("FileList -->"+ context.FileListN);

//String FileListArray[] = context.FileList.split(",");
int listsize = FileList.size();
int i;

globalMap.put("SizeofList", listsize);
globalMap.put("FileList", FileList);

System.out.println("FileList -->"+ globalMap.get("FileList"));
System.out.println("Size of List -->"+ globalMap.get("SizeofList"));

for(i=0;i < listsize; i++){
String filenm = ((List<String>)globalMap.get("FileList")).get(i);
System.out.println("Processing File -->"+ filenm);
Until_ConvertXMLtoJSON_V1.XmltoJson(context.FileDir, filenm);
}

tLoop ==>

While loop

Declaration == int i=0

condition =i <((Integer)globalMap.get("SizeofList"))

Iteration =i++

Let me know if i am missing anything!

Re: Not able to iterate files in Talend Spark using tLoop

Anonymous — Wed, 26 Sep 2018 08:56:03 GMT

Hello,

tLoop component in spark job has the same feature of the DI one, except the parallelization, which does not exist on Spark.

Best regards

Sabrina

Re: Not able to iterate files in Talend Spark using tLoop

pragna_k — Wed, 26 Sep 2018 12:53:21 GMT

The tloop component did not work for me. Can you post a solution if u have
with tloop. I want same file names for input and output both.

Re: Not able to iterate files in Talend Spark using tLoop

badri-nair — Mon, 01 Jul 2019 11:51:52 GMT

HI,

Do you have any solution for the same? I have the same requirement hence the same issue .

Thanks

BAdri Nair

@xdshi wrote:

Hello,

tLoop component in spark job has the same feature of the DI one, except the parallelization, which does not exist on Spark.

Best regards

Sabrina