tXmlMap buffering

One Star

tXmlMap buffering

Hi,
We have a table with one column being an entire xml document.
In this case we want to extract part of the xml and write it to an Oracle table.
So we use an tOracleInput -> tConvertType -> tXmlMap -> tOracleOutput.
Everything works fine.
But if we run the job with a larger number of records ( +- 150 000) it takes a long time because
before the tXmlMap all xml's are buffered/cached before they go the insert section.
Is there a possibility to let this run in parallel or without buffering.
So read xml and immediately insert in oracle and then the next record instead of buffering all the xml's and then process all the inserts.
Grtz,
Bart
Moderator

Re: tXmlMap buffering

Hi,
Did you set option "Commit every" in TalendHelpCenter:tOracleOutPut on which you can nter the number of rows to be completed before committing batches of rows together into the DB. This option ensures transaction quality (but not rollback) and, above all, better performance at execution.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: tXmlMap buffering

Yes.
'Commit every' is set to 1000.
Even if we don't use an OracleOutput and just send the output to a tLogRow the buffering/caching stays.
Moderator

Re: tXmlMap buffering

Hi,
So far, there is no "store the data on disk" option in tXMLMap(it's available in tMap).
It's welcome to open a feature jira issue.
Here is a jira issue of tXMLMap doesn't work with more data https://jira.talendforge.org/browse/TDI-29958.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: tXmlMap buffering

I don't know if that would be the solution.
What we actually would want is that the records are processed one by one.
ex.: read record from OracleInput -> tXmlMap -> tLogRow or OracleOutput
What we have now is that all records are read + buffered.
After every record is buffered the further processing runs.
So the tXmlMap doesn't give any output until everything is buffered.
I guess "store data on disk" will only prevent from memory issues.
One Star

Re: tXmlMap buffering

I too have the same problem Smiley Sad.. Can anyone please help.

Re: tXmlMap buffering

As a workaround, you could iterate over your records which would force single-row operations. In general this will be much slower than processing records as a batch, but it should prevent the unwanted buffering behavior from tXMLMap.
please see screenshot below for reference.