TDM multiple CSV inputs

CSen — Sat, 16 Nov 2024 04:24:19 GMT

Hello,

I recently tried to generate a complex structured format (GS1 format in JSON or XML) from a database. The fact the output format is quite complex and pre-exist on TDM samples data drive me to use the data mapper but there is no supported connectors for this database (informix). In the following descripion i'm using 6.2.1 version of Talend but had tried to do it in 7.2.1 without success.

To create this complex structure I need to make multiple lookups from the database as it concern data from several entities.

So I though that I could generate flat files (CSV) from informix and then map multiples CSV files to the complex JSON, using the ReadURL function that is described here : https://help.talend.com/reader/njNNGkCowThcKR6notunrw/Q7rNzUuyhHhMMfjNxPvvXA

To begin with, I tried with simple structures based on these files :

file1 (first input)

column1;column2;column3
1;"example here";500
2;"second example";834.12

file2 (second input)

column1;column1_ref
"Something";1
"Something else";2
"Again";2

file3 (output example)

{
	"item_id": 2,
	"name": "second example",
	"price": 834.12,
	"tags": [
		"Something else",
		"Again"
	]
}

In the TDM, I created a structure that include the file1 and file2 structures which look like this :

multi-input structure

And here the problem begin, when I try to "Show Sample" this structure here is what I got (XML Representation):

<Flat:ROOT>
  <Flat:file1>
    <Flat:row>
      <Flat:id>1</Flat:id>
      <Flat:name>example here</Flat:name>
      <Flat:price>50010
2;</Flat:price>
    </Flat:row>
    <Flat:row>
      <Flat:id>second example</Flat:id>
      <Flat:name>
        <Flat:file2>
          <Flat:row>
            <Flat:description>Something</Flat:description>
            <Flat:item_id>1</Flat:item_id>
          </Flat:row>
          <Flat:row>
            <Flat:description>Something else</Flat:description>
            <Flat:item_id>2</Flat:item_id>
          </Flat:row>
          <Flat:row>
            <Flat:description>Again</Flat:description>
            <Flat:item_id>2</Flat:item_id>
          </Flat:row>
        </Flat:file2>
      </Flat:name>
    </Flat:row>
  </Flat:file1>
</Flat:ROOT>

instead of :

<Flat:ROOT>
  <Flat:file1>
    <Flat:row>
      <Flat:id>1</Flat:id>
      <Flat:name>example here</Flat:name>
      <Flat:price>50010</Flat:price>
    </Flat:row>
    <Flat:row>
      <Flat:id>2</Flat:id>
      <Flat:name>second example</Flat:name>
      <Flat:price>834.12</Flat:price>
    </Flat:row>
  </Flat:file1>
  <Flat:file2>
    <Flat:row>
      <Flat:description>Something</Flat:description>
      <Flat:item_id>1</Flat:item_id>
    </Flat:row>
    <Flat:row>
      <Flat:description>Something else</Flat:description>
      <Flat:item_id>2</Flat:item_id>
    </Flat:row>
    <Flat:row>
      <Flat:description>Again</Flat:description>
      <Flat:item_id>2</Flat:item_id>
    </Flat:row>
  </Flat:file2>
</Flat:ROOT>

As well as an error, due to column shift :

error on multi-input structure

Can somebody help me resolving this issue ? It seems that the "price" column which is a float is taking the line terminator (LF) and the next row "id".

Notice that this is not happening on the file1 structure :

file1 structure

Re: TDM multiple CSV inputs

Anonymous — Thu, 24 Oct 2019 09:42:40 GMT

Hello,

Could you please file a jira issue and attach your job? Our TDM experts are going to investigate this issue by checking your job and TDM resources.

Best regards

Sabrina

Re: TDM multiple CSV inputs

CSen — Wed, 30 Oct 2019 17:37:34 GMT

Hello,

I created a JIRA issue : https://jira.talendforge.org/browse/TDM-7510
And published example of TDM resources needed to reproduce.

Best regards,

Re: TDM multiple CSV inputs

Anonymous — Thu, 31 Oct 2019 09:08:27 GMT

Hello @CSen

Here is a comment from our TDM expert in your jira issue:https://jira.talendforge.org/browse/TDM-7510.

Could you please take a look at it? Feel free to add your comment in it.

Best regards

Sabrina

Re: TDM multiple CSV inputs

CSen — Thu, 31 Oct 2019 13:36:25 GMT

Hello,

Yes I did, it's working fine with the solution provided.
Just by clearing the "multi-input" structure representations (that was set to Flat) and set all sub-structures to Flat as they were supposed to be.

Thanks for the quick reply,

topic TDM multiple CSV inputs in Talend Studio

TDM multiple CSV inputs

Re: TDM multiple CSV inputs

Re: TDM multiple CSV inputs

Re: TDM multiple CSV inputs

Re: TDM multiple CSV inputs