Utilising multi-cores while fetching data from PostgresSql(EDB)

Anonymous — Sat, 16 Nov 2024 09:48:30 GMT

Hi,

We are fetching data from PostgresSql(EDB). Having two separate servers one for DB and ETL. While a talend job was called in ETL server, query inside tPostgressqlInput fetch the data using multiple cores(parallel) in DB server. I mean all the cores were utilised based on parallel workers defined.

But in ETL server, only one core was utilized. Specified the cursor size to fetch data in chunk.I doubt, it was because of JDBC driver. For single query(one connection) only one thread will be used? Can't I make the make the job to use multi-threaded while fetching the data using JDBC driver?

What i assume is, data is read sequentially and parallelism works right after tPostgressqlInput, if i enable parallelism in job level. Kindly provide some suggestion, how to make use of multi-cores while transferring data from DB server.

Thanks

Re: Utilising multi-cores while fetching data from PostgresSql(EDB)

Anonymous — Thu, 18 May 2017 10:10:17 GMT

Hi
One connection open a thread, connection can't be shared between threads because each connection is also a transaction. The 'enable parallelism in job level' is for different subjobs in the same job.

Regards
Shong

topic Utilising multi-cores while fetching data from PostgresSql(EDB) in Talend Studio

Utilising multi-cores while fetching data from PostgresSql(EDB)

Re: Utilising multi-cores while fetching data from PostgresSql(EDB)