topic Re: Loading of large XML files to Postgres in Talend Studio

Loading of large XML files to Postgres

Soumya_M — Fri, 15 Nov 2024 22:15:54 GMT

Hello, I have created a talend job which extracts xml file and loads it to postgres. But I can only load files having size in kb, when i tried loading xml file (size 53mb), it is not appearing under File XML option. I need help with this.

Re: Loading of large XML files to Postgres

Anonymous — Tue, 20 Dec 2022 09:02:06 GMT

Hello,

Did you attempt to load the large xml through the File XML wizard?

Could you please upload some screenshots of job? So that we can see which components you are using in the job and the detailed settings, and then will give you some suggestions.

Best regards

Sabrina

Re: Loading of large XML files to Postgres

Soumya_M — Tue, 20 Dec 2022 10:19:57 GMT

I'm unable to fetch the xml file or xml metadata under File XML wizard. Though it gives the schema preview successfully, the xml file does not appear under File XML wizard.

Re: Loading of large XML files to Postgres

jlolling — Tue, 20 Dec 2022 19:28:00 GMT

Of course not. I would shorten the file for the purpose of getting the schema a lot. Then studio will certainly internal die because of memory problems.

Re: Loading of large XML files to Postgres

Soumya_M — Wed, 21 Dec 2022 01:51:13 GMT

How to shorten the xml file ?

Re: Loading of large XML files to Postgres

Soumya_M — Wed, 21 Dec 2022 01:52:10 GMT

I'm unable to fetch the xml file or xml metadata under File XML wizard. Though it gives the schema preview successfully, the xml file does not appear under File XML wizard.

Re: Loading of large XML files to Postgres

Anonymous — Wed, 21 Dec 2022 04:43:55 GMT

Hello,

Here is an approach that we used it before. It is reading the big XML as CSV file with tFileInputDelimited componenet and then passing data to tFileOutputXML or tAdvancedFileOutputXML which split it into small XML files.

Best regards

Sabrina

Re: Loading of large XML files to Postgres

Soumya_M — Wed, 21 Dec 2022 05:28:53 GMT

May i know how to split the resulting tFileOutputXML into small XML files

Re: Loading of large XML files to Postgres

Anonymous — Wed, 21 Dec 2022 05:38:19 GMT

Hello,

Here is an option "Split Output in several files" in Advanced settings of tFileOutputXML component.

If the output is big, you can split the output into several files, each containing the specified number of rows.

Best regards

Sabrina

Re: Loading of large XML files to Postgres

Soumya_M — Wed, 21 Dec 2022 13:24:56 GMT

I shall try it, thank you 🙂

Re: Loading of large XML files to Postgres

Anonymous — Thu, 22 Dec 2022 08:29:37 GMT

Hello,

Don't hesitate to post your issue here.

Best regards

Sabrina