https://community.qlik.com/t5/Data-Quality/tRuleSurvivorship-tMatchGroup-performance-issue/m-p/2198655#M304 <I>Hello,</I> <BR />As part of our ETL import we wanted to identify duplicates in the file. We are using tMatchgroup? and ?tRuleSurvivorship&nbsp;&nbsp;to achieve this and were successful in identifying duplicates and create a new row for the survivor for each duplicate group. <BR />While running this job on TAC, we are facing performance issue with these components. We ran a file with 2600 records and it was successful but sluggish(took 5 mins to process it). But when we run a file with 120K records, it just gets stuck on this subjob which has&nbsp;tMatchgroup? and ?tRuleSurvivorship &nbsp;and doesn't process the data at all. <BR />We cannot even set up parallelization on this sub job due to these components. After adding a level of logging we have identified that&nbsp;these components are the bottleneck. Can someone suggest how to improve the performance of these components. <BR />We are using Talend Platform for Big Data 5.5.1.r118616, the jvm parameters for this job on TAC are set to (-Xms1024M,&nbsp;-Xmx24576M) <BR />Any advice on performance improvement or way around this logic will be highly appreciated. <BR /> <I>Thanks in advance.</I> <BR /> <span class="lia-inline-image-display-wrapper" image-alt="0683p000009MB4f.png"><img src="https://community.qlik.com/t5/image/serverpage/image-id/130272i53849DF09FB26C88/image-size/large?v=v2&amp;px=999" role="button" title="0683p000009MB4f.png" alt="0683p000009MB4f.png" /></span> Thu, 16 Jul 2015 18:44:06 GMT npatel 2015-07-16T18:44:06Z https://community.qlik.com/t5/Data-Quality/tRuleSurvivorship-tMatchGroup-performance-issue/m-p/2198655#M304 <I>Hello,</I> <BR />As part of our ETL import we wanted to identify duplicates in the file. We are using tMatchgroup? and ?tRuleSurvivorship&nbsp;&nbsp;to achieve this and were successful in identifying duplicates and create a new row for the survivor for each duplicate group. <BR />While running this job on TAC, we are facing performance issue with these components. We ran a file with 2600 records and it was successful but sluggish(took 5 mins to process it). But when we run a file with 120K records, it just gets stuck on this subjob which has&nbsp;tMatchgroup? and ?tRuleSurvivorship &nbsp;and doesn't process the data at all. <BR />We cannot even set up parallelization on this sub job due to these components. After adding a level of logging we have identified that&nbsp;these components are the bottleneck. Can someone suggest how to improve the performance of these components. <BR />We are using Talend Platform for Big Data 5.5.1.r118616, the jvm parameters for this job on TAC are set to (-Xms1024M,&nbsp;-Xmx24576M) <BR />Any advice on performance improvement or way around this logic will be highly appreciated. <BR /> <I>Thanks in advance.</I> <BR /> <span class="lia-inline-image-display-wrapper" image-alt="0683p000009MB4f.png"><img src="https://community.qlik.com/t5/image/serverpage/image-id/130272i53849DF09FB26C88/image-size/large?v=v2&amp;px=999" role="button" title="0683p000009MB4f.png" alt="0683p000009MB4f.png" /></span> Thu, 16 Jul 2015 18:44:06 GMT https://community.qlik.com/t5/Data-Quality/tRuleSurvivorship-tMatchGroup-performance-issue/m-p/2198655#M304 npatel 2015-07-16T18:44:06Z https://community.qlik.com/t5/Data-Quality/tRuleSurvivorship-tMatchGroup-performance-issue/m-p/2198656#M305 Hi <A href="https://www.talendforge.org/forum/profile.php?id=179125" target="_blank" rel="nofollow noopener noreferrer">npatel,</A> <BR />Could you please report a ticket on Talend Support Portal? <BR />In this way, we can give you a remote assistance on your performance issue through support cycle with priority? <BR /> <A href="https://support.talend.com/otrs/customer.pl" target="_blank" rel="nofollow noopener noreferrer">https://support.talend.com/otrs/customer.pl</A> <BR />Best regards <BR />Sabrina Mon, 20 Jul 2015 04:34:53 GMT https://community.qlik.com/t5/Data-Quality/tRuleSurvivorship-tMatchGroup-performance-issue/m-p/2198656#M305 Anonymous 2015-07-20T04:34:53Z