topic Re: Write huge text files tfileoutputdelimited in Talend Studio

Write huge text files tfileoutputdelimited

Anonymous — Wed, 03 Jan 2018 21:55:24 GMT

Hello,

I have a need to run a query on Oracle that results more than 500k records and write to a text file. I have just used two components as below

toracleinput -> tfileoutputdelimited

The current write speed is ~250 rows/second. How can I increase the speed of data write? I have enterprise subscription and "enable parallel execution" on tfileoutputdelimited is greyed out as it seems to be available only for map/reduce jobs.

Please suggest.

Thanks,

Surya

Re: Write huge text files tfileoutputdelimited

Théo_Cap — Thu, 04 Jan 2018 09:19:00 GMT

Hello,

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different number.

Re: Write huge text files tfileoutputdelimited

Théo_Cap — Thu, 04 Jan 2018 09:19:27 GMT

Hello,

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different numbers.

Re: Write huge text files tfileoutputdelimited

TRF — Thu, 04 Jan 2018 09:36:28 GMT

Hi,

What if you don't push the result to a file or if you execute the same query from SQL (out of TOS)?

Re: Write huge text files tfileoutputdelimited

Anonymous — Thu, 04 Jan 2018 13:03:01 GMT

The problem is with your query or your database component config. I've just tested writing a million rows of 600 chars to a flat file and got a performance of 49465 rows per second written.....on a reasonably middle of the road machine. Remove your fileoutput component and add a tJavaFlex (with nothing configured.....just to see the rows per second metrics). How long does that take? Now tweak your query component and your query.

Re: Write huge text files tfileoutputdelimited

Anonymous — Tue, 09 Jan 2018 13:40:37 GMT

I was just doing a select * from Table.

Re: Write huge text files tfileoutputdelimited

Anonymous — Tue, 09 Jan 2018 13:41:12 GMT

SQL runs fine. It's just with the export speed

Re: Write huge text files tfileoutputdelimited

Anonymous — Tue, 09 Jan 2018 21:23:47 GMT

The SQL might run fine, but the Talend database component may not be configured adequately. As I have said, you can write thousands of rows per second to a flat file. That will not be the bottleneck. Have you tested removing the fileoutput component and just connecting to a tJavaFlex to compare the performance?

Can you show us screenshots of your job and your db component config (including the Advanced tab)?