Hi all,
I am trying to extract data from Oracle and I want to replace existing tDBConnection + tDBInput extract with Oracle SQLPlus, as the latter extracts data 3 times faster.
I have written SQLPlus script and I am able to run it in Console/ CMD.
Now the task is to run it from Talend. The problem is in following: when I try to use tSystem component to execute my SQLPlus program and use STDOUT of this program as an input to a tFileOutputDelimited it reads all data into memory and only then tries to write it to this file. This is not ok as when the amount of data is big, it fails with "Not enough memory" error.
I have tried to use Streaming as well as tFlowToIterate but it doesn't help. For example when I check the java code generated when I use tFlowToIterate, I see that it first tries to create a long vector with all rows extracted from Oracle and then loops through this vector.
I however know that if I use tDBInput + tFLowToIterate then Talend will create only one loop, where it will extract value from DB and store/ process it immediately. -- the behaviour I want to achieve
Please advise what would be the best option in my case? I can't find any good solution, and the best that comes into my mind is to write custom component.
Thank you.
