Load files in specific sequence

[sunbaj]

Hi,

I need to load files from a folder in a specific sequence. I might get more than 1 file in a day. So I need to load the file in a specific sequence only in a loop so that the oldest file loads first and the full process id executed and then and then the subsequent files are loaded one by one.

How can I achieve this. tFileList does not specify the sequence. It randomly picks the file in any sequence but I need to load the files in a specific sequence. 

I have loaded the file properties in a table and now I would pull the file name and them loop through it and process it one by one. 

What would be the best way to achieve this.

Thanks

Sunil

[manodwhb]

@sunbaj, you need to leverage the order by and order by action properties in tfilelist basic settings.

[sunbaj]

Thanks for the quick response but that would be very risky. Of all the options available I can only order it by modified date but what if for some reason I have the modified date wrong. I would like to sort it by name but that cannot be done as the file name is in the below format Rpt 2 May 2020, Rpt 20 May 2020 etc..

So is there any other way to achieve this.

I have loaded the files in a table and added a column which has the date field so that the file can be sorted by the date but not sure how to pass the file name in the job in a loop using a sql query

[billimmer · Accepted solution]

Then you are almost there.  Just read your table with a tdbinput component and connect it to a tFlowToIterate.  Then connect an "iterate" to the next step of your job.  The example on this page should help:

https://help.talend.com/reader/wDRBNUuxk629sNcI0dNYaA/WqJCkTE5~5DDaSFvtNXZnQ

Scenario: Transforming data flow to a list

The following scenario describes a Job that reads a list of files from a defined input file, iterates on each of the files and displays their content row by row on the Run console.

[sunbaj]

I tried to implement the way as mentioned above. However not sure why I get the below error. When I copy and paste the path in file explorer the file opens up.

xception in component tFileInputExcel_1 (childjob)
java.io.FileNotFoundException: C:\Users\su\Desktop\Z\File_To_Process\Revenue 2 Mar 2020 (The system cannot find the file specified)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at zrr.childjob_0_1.childjob.tDBInput_2Process(childjob.java:3389)
at zrr.childjob_0_1.childjob.runJobInTOS(childjob.java:5074)
at zrr.childjob_0_1.childjob.main(childjob.java:4908)

[billimmer]

Does the file exist?  Try the tFileExist.  Is the file found?

"C:\Users\su\Desktop\Z\File_To_Process\Revenue 2 Mar 2020" 

[sunbaj]

Yes. File exists. When I copy and paste the path in file explorer it opens that file but in Talend it gives error

[sunbaj]

Attaches is the screen shot.

Not sure why the file component fails.

This is the file path that I pass in file and also in tFileExist1

"C:/Users/su/Desktop/Z/File_To_Process/"+((String)globalMap.get("Name_of_File"))

Untitled.png

[billimmer]

Add a tJava before the tfileexists with this:

System.out.println("C:/Users/su/Desktop/Z/File_To_Process/"+((String)globalMap.get("Name_of_File")));

What does it show in the console?  Is the path and filename correct?

[billimmer]

Also, when using the tFileExist, you connect to your ZRR_FILE with an "if" connector, not a OnComponentOk.  See the tFileExist help.