Optimize data loading into Database

MKAPOOR1596038160 — Mon, 24 Aug 2020 18:23:46 GMT

I would like to optimize my data loading strategy with Talend. My scenario is as follows.

I am doing extraction and Transformation of data using Talend and generating files with suffix as time( yyyy-MM-dd_HH:mm:ss) because my collection and transformation frequency is in range of minutes ( 5, 10, 20, 30) for different flows of data. Currently i am having same frequency of data loading as it is for data extraction and transformation which is generating small size files and DB remains loaded. So i prefer to do loading every 3 hours or may be different but i am out of ideas how to play with small files generated time based by Extraction & Transformation.

For example

I could use append with intFileoutputDelimited component with suffix as (yyyy-MM-dd-HH) so that one file gets generated every hour but i am not sure how to ensure that :

My Talend loader which should run every 3 hours should only process past all hours files which have been completed and Not process the current hour file.

I also thought of renaming the files currently generating or to keep them to another directory but not sure how to do it fetch them on time basis?

I would appreciate if anyone can help on this.

topic Optimize data loading into Database in Talend Studio

Optimize data loading into Database