topic Re: Load files in specific sequence in Talend Studio

Load files in specific sequence

sunbaj — Sat, 16 Nov 2024 02:28:11 GMT

Hi,

I need to load files from a folder in a specific sequence. I might get more than 1 file in a day. So I need to load the file in a specific sequence only in a loop so that the oldest file loads first and the full process id executed and then and then the subsequent files are loaded one by one.

How can I achieve this. tFileList does not specify the sequence. It randomly picks the file in any sequence but I need to load the files in a specific sequence.

I have loaded the file properties in a table and now I would pull the file name and them loop through it and process it one by one.

What would be the best way to achieve this.

Thanks

Sunil

Re: Load files in specific sequence

manodwhb — Tue, 12 May 2020 10:41:47 GMT

@sunbaj, you need to leverage the order by and order by action properties in tfilelist basic settings.

Re: Load files in specific sequence

sunbaj — Tue, 12 May 2020 10:54:49 GMT

Thanks for the quick response but that would be very risky. Of all the options available I can only order it by modified date but what if for some reason I have the modified date wrong. I would like to sort it by name but that cannot be done as the file name is in the below format Rpt 2 May 2020, Rpt 20 May 2020 etc..

So is there any other way to achieve this.

I have loaded the files in a table and added a column which has the date field so that the file can be sorted by the date but not sure how to pass the file name in the job in a loop using a sql query

Re: Load files in specific sequence

billimmer — Tue, 12 May 2020 15:08:30 GMT

Then you are almost there. Just read your table with a tdbinput component and connect it to a tFlowToIterate. Then connect an "iterate" to the next step of your job. The example on this page should help:

https://help.talend.com/reader/wDRBNUuxk629sNcI0dNYaA/WqJCkTE5~5DDaSFvtNXZnQ

Scenario: Transforming data flow to a list

The following scenario describes a Job that reads a list of files from a defined input file, iterates on each of the files and displays their content row by row on the Run console.

Re: Load files in specific sequence

sunbaj — Wed, 13 May 2020 14:55:28 GMT

I tried to implement the way as mentioned above. However not sure why I get the below error. When I copy and paste the path in file explorer the file opens up.

xception in component tFileInputExcel_1 (childjob)
java.io.FileNotFoundException: C:\Users\su\Desktop\Z\File_To_Process\Revenue 2 Mar 2020 (The system cannot find the file specified)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at zrr.childjob_0_1.childjob.tDBInput_2Process(childjob.java:3389)
at zrr.childjob_0_1.childjob.runJobInTOS(childjob.java:5074)
at zrr.childjob_0_1.childjob.main(childjob.java:4908)

Re: Load files in specific sequence

billimmer — Wed, 13 May 2020 15:02:06 GMT

Does the file exist? Try the tFileExist. Is the file found?

"C:\Users\su\Desktop\Z\File_To_Process\Revenue 2 Mar 2020"

Re: Load files in specific sequence

sunbaj — Wed, 13 May 2020 15:12:56 GMT

Yes. File exists. When I copy and paste the path in file explorer it opens that file but in Talend it gives error

Re: Load files in specific sequence

sunbaj — Wed, 13 May 2020 15:30:53 GMT

Attaches is the screen shot.

Not sure why the file component fails.

This is the file path that I pass in file and also in tFileExist1

"C:/Users/su/Desktop/Z/File_To_Process/"+((String)globalMap.get("Name_of_File"))

Untitled.png

Re: Load files in specific sequence

billimmer — Wed, 13 May 2020 16:07:33 GMT

Add a tJava before the tfileexists with this:

System.out.println("C:/Users/su/Desktop/Z/File_To_Process/"+((String)globalMap.get("Name_of_File")));

What does it show in the console? Is the path and filename correct?

Re: Load files in specific sequence

billimmer — Wed, 13 May 2020 16:12:12 GMT

Also, when using the tFileExist, you connect to your ZRR_FILE with an "if" connector, not a OnComponentOk. See the tFileExist help.

Re: Load files in specific sequence

sunbaj — Wed, 13 May 2020 16:57:15 GMT

Not been able to resolve it. So near to solution yet so far from resolving it

Re: Load files in specific sequence

sunbaj — Wed, 13 May 2020 17:17:52 GMT

Finally able to resolve it. I had to append the file type which I was not doing. So now finally I have been able to run it in a loop.

Thanks