topic Re: Probem: Talend is writing multiple xml rows into 1 xml file in Talend Studio

Probem: Talend is writing multiple xml rows into 1 xml file

Anonymous — Sat, 16 Nov 2024 02:23:30 GMT

Hello,

I'm currently parsing a xml file containing multiple <Case> tags and creating 1 new xml file for each <Case> tag found.

This works good so far, 1 row goes into the tXmlMap 29 rows go out (29 cases), but then when I want to write the xml files using a tFileOutputXml, I end up with 29 files (good) which all contain each and everyone of the 29 new files (bad).

So on one side the FileOutput create 1 File for each row, but on the other hand seems to add all rows into everyone of these files and I cannot understand why.

Has anyone encountered something like this?

Re: Probem: Talend is writing multiple xml rows into 1 xml file

TRF — Tue, 19 May 2020 15:17:08 GMT

You should share your job design.

Anyway, after tXmlMap you should have something like:

tFlowToIterate-->tJava(build next filename)-->tFixedFlowInput(renew the data flow 1 row)--> tFileOutputXML

Re: Probem: Talend is writing multiple xml rows into 1 xml file

Anonymous — Tue, 19 May 2020 20:43:29 GMT

Thanks, I dont quite understand you solution,

so here are some Screenshots to provide more details on the problem.

First the structure of the job itself:

Then the Mapping (Note the source XML contains multiple, for example 29 cases)

And the Xml FileOutput (I would expect 29 files each containing 1 Case, instead i'm getting 29 files each containing 29 cases).

Can anyone give advice to why this happens or how I can fix this?

Re: Probem: Talend is writing multiple xml rows into 1 xml file

TRF — Tue, 19 May 2020 21:05:46 GMT

Questions:
- what's in the lookup?
- where do the context variables come from (as it seems this is the only dynamic part of filename)?
- can you share a capture with rows count (after the job has finished)?

Re: Probem: Talend is writing multiple xml rows into 1 xml file

Anonymous — Tue, 19 May 2020 21:28:56 GMT

The Lookup and the context Variables could be ommitted, the lookup only holds specifications etc. that get pulled into the mapping via an id.

The Context Variables likewise get pulled from the same source xml to build the fileName for each row (which works correctly), I have also tried this with a static filename (ending up with one file contianing all 29 cases).

Here is a Screenshot with Rowcount.

Re: Probem: Talend is writing multiple xml rows into 1 xml file

TRF — Tue, 19 May 2020 23:16:18 GMT

Try the suggested design
tXmlMap-->tFloToIterate(here, 1 created var will contain the target filename)--(iterate)-->tFixedFlowInput(generate the flow for 1 case)-->tFileOutputXML
This is a common design and it works when you want as many output files than rows in the input flow

Re: Probem: Talend is writing multiple xml rows into 1 xml file

Anonymous — Wed, 20 May 2020 07:21:19 GMT

Okay that sounds promising, thank you.

But could you elaborate on how to generate the flow in the FixedFlowInput?
Would I need to tempSave the row / case before the FlowToIterate or how can I access the rows coming out of the txmlMap in the FixedFlowInput Component?

Thanks

Re: Probem: Talend is writing multiple xml rows into 1 xml file

TRF — Wed, 20 May 2020 08:10:05 GMT

When you use tFlowToIterate, each field from the input flow is transformed into a global variable.

By default the variable name is the same as the field name preceded by the flow name such as "row1.CaseXml".

To initiate a new data flow from this or these global variables, you need to use tFixedFlowInput (or tIterateToFlow) in which you start by defining the desired schema, for example 1 field called CaseXml datatype Document. Then, assign each field a value issued from the corresponding global variable.

Here is how it could look like:

Here is how the job design:

Re: Probem: Talend is writing multiple xml rows into 1 xml file

Anonymous — Wed, 20 May 2020 08:19:11 GMT

Thanks alot for that explanation.

Re: Probem: Talend is writing multiple xml rows into 1 xml file

Anonymous — Wed, 20 May 2020 09:23:00 GMT

I tried to implement this, but I ended up with 29 files all containing 1 of the 29 cases.

So some improvement but somehow not working as expected.

Can you maybe provide some more advice?