I would like to optimize my data loading strategy with Talend. My scenario is as follows.
I am doing extraction and Transformation of data using Talend and generating files with suffix as time( yyyy-MM-dd_HH:mm:ss) because my collection and transformation frequency is in range of minutes ( 5, 10, 20, 30) for different flows of data. Currently i am having same frequency of data loading as it is for data extraction and transformation which is generating small size files and DB remains loaded. So i prefer to do loading every 3 hours or may be different but i am out of ideas how to play with small files generated time based by Extraction & Transformation.
For example
I could use append with intFileoutputDelimited component with suffix as (yyyy-MM-dd-HH) so that one file gets generated every hour but i am not sure how to ensure that :
- My Talend loader which should run every 3 hours should only process past all hours files which have been completed and Not process the current hour file.
I also thought of renaming the files currently generating or to keep them to another directory but not sure how to do it fetch them on time basis?
I would appreciate if anyone can help on this.
