topic TDM Mapper - need to input multiple Json streams in Talend Studio

TDM Mapper - need to input multiple Json streams

DaveTaylor — Thu, 27 Jul 2017 23:10:19 GMT

I have a TDI job that calls an tHMap, and passes into it for input; a JsonDataStream; and generates output in Xml.

The tHMap works fine.

I have a new requirement, where I need to add an additional 1 or possible 2 different Json lookup streams; to be used with the main JsonDataStream; in doing some mapping for the Xml output.

It seems that tHMap only allows 1 input.
I cannot use tXmlMap; because my output Xml structure is highly complex xml with many child of child group and loops; of which tXmlMap doesn't support.

What is the best practice for having additional lookup Json streams along with my main data Json stream into my tHMap?

Should I combine the lookup jsons 'into' my main data Json stream; before loading the tHMap?

Just looking for the best practice for how to do this. Haven't found any documentation on merging or combining different json files into one; or reading lookup files from within a Mapper.

Re: TDM Mapper - need to input multiple Json streams

Anonymous — Fri, 28 Jul 2017 07:09:04 GMT

Hi,

Indeed there is currently a feature request for this requirement:
https://jira.talendforge.org/browse/TDM-5377

Nontheless, I can suggest a workaround:
-You can create your structures for all your input files (In my example, I have two input files):

and

-Then in a map you can combine your structures. At root level, you have to make sure you have no I/O function

-For each of your input structure, you assign a READURL function to read you input file

-In your job, your tHmap will have no input flow

The weakness of this approach is that you will have to write down your input files on the file system.

Let me know if it meets your requirements or you need clarifications.

Eric

Re: TDM Mapper - need to input multiple Json streams

DaveTaylor — Fri, 28 Jul 2017 15:15:16 GMT

Eric,
Thanks for the reply and the suggested work-around.

In my usecase, I am not reading my Json inputs from file; they are going to be served up via JSP-APIs, and will be Json Streams.

I haven't used the 'ReadUrl' function; do you know if it is able to also read from a stream as opposed to a filesystem input?

If not, is there an alternative to join Json streams?

My other thought I am going to try; is inputting the Jsons into a tMap/or tXmlMap component, (that can receive multiple inputs), and attempt to map them to a single json outfrom from that map component....just haven't gotten it wired up to see if that approach will work.

Re: TDM Mapper - need to input multiple Json streams

Anonymous — Fri, 28 Jul 2017 17:26:18 GMT

No ReadURL only enable reading fils from a file system.

Eric

Re: TDM Mapper - need to input multiple Json streams

DaveTaylor — Fri, 28 Jul 2017 18:18:04 GMT

Thanks Eric.

For now I am trying your suggested approach above.

I have two input Json files.

I have made a structure for each(POs and Orders); and a third combined structure(Multi), that contains them both.

I am missing something from your 3 and 4 example of the Multi-Structure.

When you say 'at the root level, make sure you have no I/O function.

(my root Node's IO/Database tab is empty)

However, when I select either of my sub elements in the structure under the root (POs or Orders); neither of them have an IO/Database tab to place a ReadURL expression object into.

Only the Root element of the structure has the 'IO/Database' tab available.
I cannot find any settings to disable it there; or to enable that tab on the POs or Orders Elements.

I can see in your last screencap, you do have the POs node under Root selected; and it "is" showing to have a IO/Database tab for you to venue it's ReadURL expression object into.

To restate what I did; following your example:
I have:
Structure1: POs
Structure2: Orders

Structure3: Multi (with Root and child siblings of POs and Orders underneath)

I feel like I am missing it where you say 'in a map you can combine your structures'.
When I open a new map; it will only allow me to have 1 input structure...Can't see how to combine both into the map.
I can use the Multi Structure; but when I use it as the input structure to my map; the child (POs or Orders) elements don't have the IO/Database tab available on them.

What am I missing?

Here are my screenshots:

Re: TDM Mapper - need to input multiple Json streams

Anonymous — Sat, 29 Jul 2017 06:46:14 GMT

Hi,

In your "Multi" input structure (The one that combine the individual structures) do you inherit from each structure ?

You should have something like:

Eric

READURL.zip

Re: TDM Mapper - need to input multiple Json streams

Anonymous — Sat, 29 Jul 2017 06:46:55 GMT

In my last reply, I uploaded my solution...

Re: TDM Mapper - need to input multiple Json streams

Anonymous — Thu, 10 Aug 2017 14:45:48 GMT

Or you can merge your inputs before calling tHmap (In such case remove data in I/O tab at map level).

Here is an example:

The tReplace remove feed lines and header in XML files

As I read files with tFileInputRaw, input is object type. I convert them into string with tConvertType

tsetGlovalVar store the two input in global variables

In tJavaRow, I merge the to files before calling tHmap (Adding a root element)

output_row.FILEMERGED = "<ROOT>"+((String)globalMap.get("FILE1"))
+ ((String)globalMap.get("FILE2"))+"</ROOT>";

If your inputs are Json and not files but stream, you will have to adjust slightly this process.