topic tMap with Dynamic schema in Talend Studio

tMap with Dynamic schema

Anonymous — Sat, 16 Nov 2024 09:01:31 GMT

Hi,

I am pretty new to the talend tool. I am attempting to use talend tMap component to map data from two sources. However I also require the schemas to be dynamic as this is to be done for numerous tables or files therefore we want to escape having to define schema for each and using that in a tMap. What I've come across is that the output for this job is different when running a job within talend studio and eclipse. It's pretty strange as to why this is the case since I exported the exact same code(job) and built it in eclipse IDE. If anybody has ever had to use dynamic schemas together with tMap, I'd like some pointers as to how this can be achieved or possibly point out what I could be doing incorrectly.

Re: tMap with Dynamic schema

med1 — Wed, 22 Nov 2017 14:24:13 GMT

Hello,

Could you please elaborate your cases ? I used dynamic schema and to handle the this schema i used the tjavarow.

Re: tMap with Dynamic schema

brama — Wed, 22 Nov 2017 16:11:24 GMT

Hi,
If I understand correctly, you want to load different schema to target. If you are using a licensed version there is a data type dynamic itself in tmap. You can make use of it.

Thanks,
Ramachandran B

Re: tMap with Dynamic schema

Anonymous — Thu, 23 Nov 2017 07:58:02 GMT

We have a scenario where we need to compare data from two sources say a file and table in a database and we need to do a comparison for each column in a file to each column in a database. We do this to identify any differences that may exist between the two sources of data. We have managed to achieve this by using a tMap component to compare each column/field in a table to each column/field in a file. However it's a lot of work if you're to do this for many tables/files, having to go through mapping each column to column in a tMap and end up designing many jobs that do exactly the same thing. So we now exploring the use of dynamic type for the schemas. I attached some screen shots on my first post.

Re: tMap with Dynamic schema

med1 — Thu, 23 Nov 2017 08:11:57 GMT

Hello,

I think for your case it is better to use dynamic schema.

So first try to select data from table to tmp file csv and then compare two files. If you have MDM module you can use specific component which allow to compare two files and give the distance between the columns etc..

BR,

Med

Re: tMap with Dynamic schema

Anonymous — Fri, 24 Nov 2017 08:21:19 GMT

Thanks for your assistance, it's much appreciated.

In a case of using Dynamic schema, can it work correctly for joining data using a tMap component? Because what lead me here in the first place was the problem I got using a dynamic type to map my data.

I don't know of the MDM module, from components this is all I got (please see attached screen shot). I will lookup some YouTube tutorials on it.

Regards,

MDM.PNG

Re: tMap with Dynamic schema

med1 — Fri, 24 Nov 2017 08:29:34 GMT

Hello,

So as i explained in my last post, you can insert the data from table into tmp file.

After that you can use tJavarow to parse the both objects ( Dynamic schema is like java object).

Here is the code to parse the dynamic File :

Dynamic columns = row3.dynamic;

for (int i = 0; i < columns.getColumnCount(); i++) {
DynamicMetadata columnMetadata = columns.getColumnMetadata(i);
System.out.println(columnMetadata.getName() + ": " +
columns.getColumnValue(i) + "; " + columnMetadata.getType());
}

There is an example in Talend Documentation, which explain how to handle the Dynamic schema :

https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/WX1guuvSnxZg5nBBQ26LBQ

Good luck

Re: tMap with Dynamic schema

Anonymous — Wed, 17 Jan 2018 08:37:49 GMT

Thank you so much. Your code solution works for my case.

Re: tMap with Dynamic schema

Anonymous — Sun, 01 Jul 2018 13:09:07 GMT

Hi,
We do have the same scenario of comparing dynamic schema files.Could you please share us the solution ?

Re: tMap with Dynamic schema

ravi999 — Fri, 28 Sep 2018 03:06:00 GMT

This worked for one of my usecase. Thanks!

Re: tMap with Dynamic schema

Anonymous — Mon, 21 Jan 2019 07:34:03 GMT

Could you please share the sample mapping? I could not understand, how to configure this scenario with Tmap?

Thanks in advance.

Re: tMap with Dynamic schema

KarthikGs — Thu, 09 May 2019 11:40:44 GMT

https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/WX1guuvSnxZg5nBBQ26LBQ

Re: tMap with Dynamic schema

Anonymous — Wed, 26 Jun 2019 10:04:50 GMT

Please refer to my new blog on the same topic.

http://www.rajeshsirsikar.com/dynamic-schema-in-talend-open-studio/

Thanks

Re: tMap with Dynamic schema

dm-ALSA — Sat, 06 Jun 2020 17:41:40 GMT

Hi Samu,

I am also having the same requirement to compare the 2 tables column by column.Please let us know what was done to compare the data dynamically.If possible please share the code.Thanks