tXmlMap buffering

[Anonymous]

Hi,
We have a table with one column being an entire xml document.
In this case we want to extract part of the xml and write it to an Oracle table.
So we use an tOracleInput -> tConvertType -> tXmlMap -> tOracleOutput.
Everything works fine.
But if we run the job with a larger number of records ( +- 150 000) it takes a long time because
before the tXmlMap all xml's are buffered/cached before they go the insert section.
Is there a possibility to let this run in parallel or without buffering.
So read xml and immediately insert in oracle and then the next record instead of buffering all the xml's and then process all the inserts.
Grtz,
Bart

[Anonymous]

Hi,
Did you set option "Commit every" in TalendHelpCenter:tOracleOutPut on which you can nter the number of rows to be completed before committing batches of rows together into the DB. This option ensures transaction quality (but not rollback) and, above all, better performance at execution.
Best regards
Sabrina

[Anonymous]

Yes.
'Commit every' is set to 1000.
Even if we don't use an OracleOutput and just send the output to a tLogRow the buffering/caching stays.

[Anonymous]

Hi,
So far, there is no "store the data on disk" option in tXMLMap(it's available in tMap).
It's welcome to open a feature jira issue.
Here is a jira issue of tXMLMap doesn't work with more data https://jira.talendforge.org/browse/TDI-29958.
Best regards
Sabrina

[Anonymous]

I don't know if that would be the solution.
What we actually would want is that the records are processed one by one.
ex.: read record from OracleInput -> tXmlMap -> tLogRow or OracleOutput
What we have now is that all records are read + buffered.
After every record is buffered the further processing runs.
So the tXmlMap doesn't give any output until everything is buffered.
I guess "store data on disk" will only prevent from memory issues.

[Anonymous]

I too have the same problem .. Can anyone please help.

[Anonymous]

As a workaround, you could iterate over your records which would force single-row operations. In general this will be much slower than processing records as a batch, but it should prevent the unwanted buffering behavior from tXMLMap.
please see screenshot below for reference.