Hi All,
I have attached my job workflow. I am getting file without header. I am adding the header in the file.
Now I am reading same file and applying some transformation and I need to insert data into the database table
I am making this talend job as generic.
So while reading from file I have selected the metadata col as dynamic after that I have used tjavaflex component wherer I am reading the dynamic columns and applying any transformation if neededwhich basically makes the select query for every row by taking the data from file and apply the transformation into it.
e.g select 'B1' As Emp,length(deptno) as departmentno,Lpad(empno) as empno from dual
and then I am iterating the query using the component tflowtoiterate and passing it to toracleinput in which I am passing my query and then load that query data into one table.
Now my flow is working fine. For 11k records to insert it takes 2,min since it create query for each row it takes too much of time to load the data.
We have 500k data in that case it will take too much of time.
can anyone please help me to optimise this job.
