topic tXmlMap buffering in Talend Studio

tXmlMap buffering

Anonymous — Mon, 30 Jun 2014 08:40:43 GMT

Hi,
We have a table with one column being an entire xml document.
In this case we want to extract part of the xml and write it to an Oracle table.
So we use an tOracleInput -> tConvertType -> tXmlMap -> tOracleOutput.
Everything works fine.
But if we run the job with a larger number of records ( +- 150 000) it takes a long time because
before the tXmlMap all xml's are buffered/cached before they go the insert section.
Is there a possibility to let this run in parallel or without buffering.
So read xml and immediately insert in oracle and then the next record instead of buffering all the xml's and then process all the inserts.
Grtz,
Bart

Re: tXmlMap buffering

Anonymous — Wed, 02 Jul 2014 07:21:09 GMT

Hi,
Did you set option "Commit every" in TalendHelpCenter:tOracleOutPut on which you can nter the number of rows to be completed before committing batches of rows together into the DB. This option ensures transaction quality (but not rollback) and, above all, better performance at execution.
Best regards
Sabrina

Re: tXmlMap buffering

Anonymous — Wed, 02 Jul 2014 09:06:02 GMT

Yes.
'Commit every' is set to 1000.
Even if we don't use an OracleOutput and just send the output to a tLogRow the buffering/caching stays.

Re: tXmlMap buffering

Anonymous — Wed, 02 Jul 2014 09:22:54 GMT

Hi,
So far, there is no "store the data on disk" option in tXMLMap(it's available in tMap).
It's welcome to open a feature jira issue.
Here is a jira issue of tXMLMap doesn't work with more data https://jira.talendforge.org/browse/TDI-29958.
Best regards
Sabrina

Re: tXmlMap buffering

Anonymous — Wed, 02 Jul 2014 10:51:43 GMT

I don't know if that would be the solution.
What we actually would want is that the records are processed one by one.
ex.: read record from OracleInput -> tXmlMap -> tLogRow or OracleOutput
What we have now is that all records are read + buffered.
After every record is buffered the further processing runs.
So the tXmlMap doesn't give any output until everything is buffered.
I guess "store data on disk" will only prevent from memory issues.

Re: tXmlMap buffering

Anonymous — Wed, 12 Nov 2014 17:15:00 GMT

I too have the same problem .. Can anyone please help.

Re: tXmlMap buffering

Anonymous — Wed, 12 Nov 2014 18:29:04 GMT

As a workaround, you could iterate over your records which would force single-row operations. In general this will be much slower than processing records as a batch, but it should prevent the unwanted buffering behavior from tXMLMap.
please see screenshot below for reference.