Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
1 Solution
Accepted Solutions
This is a very common task that is not super easy to implement in Talend.
Please have a look at my example job below and let me know if this helps you, or if I can assist further
To accomplish getting the newest file, we will get a list of files then get the properties for each of them. We will then sort the file properties by "mtime" or the last modified time and then grab the oldest for further processing.
1) tFileList: this component is configured to look for files that start with my chosen string
2) tFileProperties: this component will retrieve the properties for each file.
3) tBufferOutput: this component will store the file properties in memory so we can sort them once we've got info on all the files.
4) tBufferInput: this component will read from the buffer we populated with file property information
5) tSortRow: this component will sort the files by mtime descending (meaning the oldest file will be first in the list)
6) tSampleRow: this component is how we grab only the first row coming out of tSortRow
Please have a look at my example job below and let me know if this helps you, or if I can assist further
To accomplish getting the newest file, we will get a list of files then get the properties for each of them. We will then sort the file properties by "mtime" or the last modified time and then grab the oldest for further processing.
1) tFileList: this component is configured to look for files that start with my chosen string
2) tFileProperties: this component will retrieve the properties for each file.
3) tBufferOutput: this component will store the file properties in memory so we can sort them once we've got info on all the files.
4) tBufferInput: this component will read from the buffer we populated with file property information
5) tSortRow: this component will sort the files by mtime descending (meaning the oldest file will be first in the list)
6) tSampleRow: this component is how we grab only the first row coming out of tSortRow
Hi Rajesh
Let me first ensure, if I've captured your requirements correctly:
1. Your source folder is fixed.
2. You intend to run your job every 2 hrs.
3. On every execution, you wish to pick the latest file (irrespective of its name).
Your confirmation would help formulate a solution in a better way.
MathurM
Let me first ensure, if I've captured your requirements correctly:
1. Your source folder is fixed.
2. You intend to run your job every 2 hrs.
3. On every execution, you wish to pick the latest file (irrespective of its name).
Your confirmation would help formulate a solution in a better way.
MathurM
This is a very common task that is not super easy to implement in Talend.
Please have a look at my example job below and let me know if this helps you, or if I can assist further
To accomplish getting the newest file, we will get a list of files then get the properties for each of them. We will then sort the file properties by "mtime" or the last modified time and then grab the oldest for further processing.
1) tFileList: this component is configured to look for files that start with my chosen string
2) tFileProperties: this component will retrieve the properties for each file.
3) tBufferOutput: this component will store the file properties in memory so we can sort them once we've got info on all the files.
4) tBufferInput: this component will read from the buffer we populated with file property information
5) tSortRow: this component will sort the files by mtime descending (meaning the oldest file will be first in the list)
6) tSampleRow: this component is how we grab only the first row coming out of tSortRow
Please have a look at my example job below and let me know if this helps you, or if I can assist further
To accomplish getting the newest file, we will get a list of files then get the properties for each of them. We will then sort the file properties by "mtime" or the last modified time and then grab the oldest for further processing.
1) tFileList: this component is configured to look for files that start with my chosen string
2) tFileProperties: this component will retrieve the properties for each file.
3) tBufferOutput: this component will store the file properties in memory so we can sort them once we've got info on all the files.
4) tBufferInput: this component will read from the buffer we populated with file property information
5) tSortRow: this component will sort the files by mtime descending (meaning the oldest file will be first in the list)
6) tSampleRow: this component is how we grab only the first row coming out of tSortRow
Hi JohnGarrettMartin, I feel with your above solution, we kind of drifted away a bit from the original problem.
Hi Rajesh,
I would suggest you try an approach on the lines of the job shown below.
Here,
1. We first create a start flag (assigning it a value, say 'T')
2. Using tFileList component, we iteratively extract all the files from the source folder. This component, itself allows us to sort the order of the files. We can sort the files on 'modified date', & also arrange them in 'ASC or DESC' order. In present case, we choose 'DESC.
3. Further on, we arrange to iteratively process each of the file based on a 'IF' condition i.e. the 'FLAG' equals 'T'
4. On successful processing of the file, on a 'OnSubjobOk' link we change the 'FLAG' to say 'F'.
5. As a result, after the successful processing of the first file, the flag would be changed from 'T' to 'F'. Hence, no-more fulfilling the 'IF' condition & no further files would be processed.
This way, we can achieve the processing of only the latest file in the source folder on every execution.
hope this helps.
MathurM
Hi Rajesh,
I would suggest you try an approach on the lines of the job shown below.
Here,
1. We first create a start flag (assigning it a value, say 'T')
2. Using tFileList component, we iteratively extract all the files from the source folder. This component, itself allows us to sort the order of the files. We can sort the files on 'modified date', & also arrange them in 'ASC or DESC' order. In present case, we choose 'DESC.
3. Further on, we arrange to iteratively process each of the file based on a 'IF' condition i.e. the 'FLAG' equals 'T'
4. On successful processing of the file, on a 'OnSubjobOk' link we change the 'FLAG' to say 'F'.
5. As a result, after the successful processing of the first file, the flag would be changed from 'T' to 'F'. Hence, no-more fulfilling the 'IF' condition & no further files would be processed.
This way, we can achieve the processing of only the latest file in the source folder on every execution.
hope this helps.
MathurM
Hi
Can I get assistance from this solution? I am currently working on the same issue (to picking up data from s3 bucket based on the latest file.
