https://community.qlik.com/t5/Talend-Studio/How-to-efficiently-load-30mil-rows-from-Kafka-into-Postgresql-w/m-p/2236240#M25020 Hi Team, <BR />I have a scenario where i want to load 30million rows of data using Kafka into PostgreSQL. My job is going to be quite straightforward with few filters and tmap.&nbsp; <BR />I would like to know: <BR />1. What are the best configuration settings for TOS in such scenario? <BR />2. how can i reduce cpu/memory consumptions? <BR />3. what is the most efficient way to load huge data using TOD? Shall i do it in one go or in batches? <BR />4. And also, what could be the average time to load such huge data using TOS? <BR />Thanks in advance. <BR />Rera Sat, 16 Nov 2024 10:45:55 GMT Anonymous 2024-11-16T10:45:55Z https://community.qlik.com/t5/Talend-Studio/How-to-efficiently-load-30mil-rows-from-Kafka-into-Postgresql-w/m-p/2236240#M25020 Hi Team, <BR />I have a scenario where i want to load 30million rows of data using Kafka into PostgreSQL. My job is going to be quite straightforward with few filters and tmap.&nbsp; <BR />I would like to know: <BR />1. What are the best configuration settings for TOS in such scenario? <BR />2. how can i reduce cpu/memory consumptions? <BR />3. what is the most efficient way to load huge data using TOD? Shall i do it in one go or in batches? <BR />4. And also, what could be the average time to load such huge data using TOS? <BR />Thanks in advance. <BR />Rera Sat, 16 Nov 2024 10:45:55 GMT https://community.qlik.com/t5/Talend-Studio/How-to-efficiently-load-30mil-rows-from-Kafka-into-Postgresql-w/m-p/2236240#M25020 Anonymous 2024-11-16T10:45:55Z https://community.qlik.com/t5/Talend-Studio/How-to-efficiently-load-30mil-rows-from-Kafka-into-Postgresql-w/m-p/2236241#M25021 Hi Rera, <BR /> <BLOCKQUOTE> <TABLE border="1"> <TBODY> <TR> <TD>I have a scenario where i want to load 30million rows of data using Kafka into PostgreSQL. My job is going to be quite straightforward with few filters and tmap. </TD> </TR> </TBODY> </TABLE> </BLOCKQUOTE> <BR />tMap is <FONT size="2"><FONT face="" calibri="">cache component consuming much memory. For </FONT></FONT> <FONT size="2"><FONT face="" calibri="">a large set of data, try to store the data on disk instead of memory.</FONT></FONT> <BR />Here is an option "Use Batch Size" in tPostgresqloutput&nbsp;which is used to activate the batch mode for data processing. <BR />What does your filter look like? What's your current row rate(rows/s)? Is it a normal speed? <BR /> Please take a look at document about: <A href="https://community.qlik.com/s/article/ka03p0000006EZuAAM" target="_blank">TalendHelpCenter:Exception outOfMemory</A> <BR />Best regards <BR />Sabrina Fri, 04 Mar 2016 07:34:03 GMT https://community.qlik.com/t5/Talend-Studio/How-to-efficiently-load-30mil-rows-from-Kafka-into-Postgresql-w/m-p/2236241#M25021 Anonymous 2016-03-04T07:34:03Z