TDM multiple CSV inputs

[CSen]

Hello,

I recently tried to generate a complex structured format (GS1 format in JSON or XML) from a database. The fact the output format is quite complex and pre-exist on TDM samples data drive me to use the data mapper but there is no supported connectors for this database (informix). In the following descripion i'm using 6.2.1 version of Talend but had tried to do it in 7.2.1 without success.

To create this complex structure I need to make multiple lookups from the database as it concern data from several entities.

So I though that I could generate flat files (CSV) from informix and then map multiples CSV files to the complex JSON, using the ReadURL function that is described here : https://help.talend.com/reader/njNNGkCowThcKR6notunrw/Q7rNzUuyhHhMMfjNxPvvXA

To begin with, I tried with simple structures based on these files :

file1 (first input)

column1;column2;column3
1;"example here";500
2;"second example";834.12

file2 (second input)

column1;column1_ref
"Something";1
"Something else";2
"Again";2

file3 (output example)

{
	"item_id": 2,
	"name": "second example",
	"price": 834.12,
	"tags": [
		"Something else",
		"Again"
	]
}

In the TDM, I created a structure that include the file1 and file2 structures which look like this :

multi-input structure

And here the problem begin, when I try to "Show Sample" this structure here is what I got (XML Representation):

<Flat:ROOT>
  <Flat:file1>
    <Flat:row>
      <Flat:id>1</Flat:id>
      <Flat:name>example here</Flat:name>
      <Flat:price>50010
2;</Flat:price>
    </Flat:row>
    <Flat:row>
      <Flat:id>second example</Flat:id>
      <Flat:name>
        <Flat:file2>
          <Flat:row>
            <Flat:description>Something</Flat:description>
            <Flat:item_id>1</Flat:item_id>
          </Flat:row>
          <Flat:row>
            <Flat:description>Something else</Flat:description>
            <Flat:item_id>2</Flat:item_id>
          </Flat:row>
          <Flat:row>
            <Flat:description>Again</Flat:description>
            <Flat:item_id>2</Flat:item_id>
          </Flat:row>
        </Flat:file2>
      </Flat:name>
    </Flat:row>
  </Flat:file1>
</Flat:ROOT>

instead of :

<Flat:ROOT>
  <Flat:file1>
    <Flat:row>
      <Flat:id>1</Flat:id>
      <Flat:name>example here</Flat:name>
      <Flat:price>50010</Flat:price>
    </Flat:row>
    <Flat:row>
      <Flat:id>2</Flat:id>
      <Flat:name>second example</Flat:name>
      <Flat:price>834.12</Flat:price>
    </Flat:row>
  </Flat:file1>
  <Flat:file2>
    <Flat:row>
      <Flat:description>Something</Flat:description>
      <Flat:item_id>1</Flat:item_id>
    </Flat:row>
    <Flat:row>
      <Flat:description>Something else</Flat:description>
      <Flat:item_id>2</Flat:item_id>
    </Flat:row>
    <Flat:row>
      <Flat:description>Again</Flat:description>
      <Flat:item_id>2</Flat:item_id>
    </Flat:row>
  </Flat:file2>
</Flat:ROOT>

As well as an error, due to column shift : 

error on multi-input structure

Can somebody help me resolving this issue ? It seems that the "price" column which is a float is taking the line terminator (LF) and the next row "id".

Notice that this is not happening on the file1 structure : 

file1 structure

[Anonymous]

Hello,

Could you please file a jira issue and attach your job? Our TDM experts are going to investigate this issue by checking your job and TDM resources.

Best regards

Sabrina

[CSen]

Hello,

I created a JIRA issue : https://jira.talendforge.org/browse/TDM-7510
And published example of TDM resources needed to reproduce.

Best regards,

[Anonymous · Accepted solution]

Hello @CSen 

Here is a comment from our TDM expert in your jira issue:https://jira.talendforge.org/browse/TDM-7510.

Could you please take a look at it? Feel free to add your comment in it.

Best regards

Sabrina

[CSen]

Hello,

Yes I did, it's working fine with the solution provided.
Just by clearing the "multi-input" structure representations (that was set to Flat) and set all sub-structures to Flat as they were supposed to be.

Thanks for the quick reply,