Jobs getting stuck while loading large data tables to target

[Anonymous]

Hi Experts,

I am using Talend Open Studio for Data Integration Version: 6.3.1. I am using Open Source version. I am trying to load data from a postgres table (10 million records) to a postgres table in my Target database. When I trigger the job, I only see that the job started, but it is not loading any data to my target table and it is not even failing. Even if I try to load the same to a flat file, even then it is not starting.

Can someone please help.

[Anonymous]

Hello @suvi 

Can you post a screenshot of the output components configuration window?or the job itself?

[Anonymous]

Hi Snishtala,

Thanks for the response. I have attached the screenshot of my job. There is nothing in my job. It is just dump of data from table to a flat file.

Talend issue.png

[Anonymous]

@suvi - What is the error on input component?

Is it related to query given in the component? Can you try putting limit on your query to see if its loading data to file at all?

[Anonymous]

@snishtala Yes on applying the filter at SQL query level, I am able to insert data to Table as well as file, but if I want to insert the entire data in one flow, its not happening.

Is there something that can be done to achieve it in one go. I tried increasing the batch job size and also the commit limit but still facing the issue.

The reason for getting it done in one go is, some of the tables are having around 60M records and if I split by applying filter, I will have to create multiple flows.

[Anonymous]

@suvi - Okay in that case what is the batch size you are using in output component?

And also are you using cursor in DB input component in advanced settings?

[Anonymous]

@snishtala I tried different batch size 1000, 10000, 100000 and same I tried for commit size.

No I have not tried the cursor from the Input component.

[Anonymous]

@suvi - Okay rather than batch try using the extended insert in input component(set it starting from 1000) and leave the commit at 10000. And also try putting the cursor value same as whatever you put for extended insert. 

[Anonymous]

@snishtala I tried to look for the "Extended Input" option but I am not able to locate it. Could you please let me know where or how to do that.

I am pretty new to Talend tool.

[Anonymous]

@snishtala sry typo "Extended Insert"