multischema multi level xml data filter Talend data mapper

[Rajender]

Hi  there

I have a multi schema and multi level XML.  File has a tag originSource at starting loop and then each multi schema and Level has a tag SourceID.  I don't want data to load into my tables if it has a source "MySource" and any of SourceID in multi schema and level is empty. But if Source it is "YourSource" and multi schema and level sourceID is null then data should be loaded successfully.

Is it possible to filter data using Talend Data Mapper.  I tried write filter condition on validate tag in child node. Then It is removing only that particular data. My expectation is to remove whole hierarchy in such case.  

Any other approach to handle it is welcome.

Thank you for your time.

Best Regards

[Anonymous]

Hi,

   If its just empty tags in XML, it will not have any impact in your final data right?

   But if you want to remove the empty XML tags also, I am not sure as I have not personally tried it. Lets hear others' opinions also.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Rajender]

Hi Nikhil

Thanks for your time. Let me put my query in this way.

Not the empty tag I want to remove. I don't want whole set of Node. For example.

<persons>

<sourceID>

<OriginSource>

  <name>

  <age>

  <address>

  <sourceID>

    <street>

    <state>

    <pin>

 </address>

 <telephone>

   <sourceID>

  <personal>

  <office>

  <fax>

</telephone>

</persons>

There can be many person in one xml. If any of the person(X) is not "MySource" and either  Address --> sourceID or telephone --> sourceID is null. Then that person(X) should not be loaded.

Best Regards

[Anonymous]

Hi,

   Could you please share the screen shot of the job you are trying to do? If you want to remove any empty records, you can try to remove them later using tFilterRow component later in the flow. 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Rajender]

Hi

There are 7 tables loaded from tXMLMap. So I am thinking to filter such records in XML itself.

Following is my job design.

fileList --> tfileInputRaw --> tHmap -> tXMLMap -> 7 output to DB tables.

Best Regards

[Anonymous]

Hi Rajender,

     I believe you are extracting the records from XML to tabular format before loading data to DB.

     You can add a tFilterRow after that component to remove the unwanted data.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Rajender]

Hi 

Yes, you are right.  

Please refer XML example mention in previous post, One person(XYZ) has 1000 address and 2000 telephone. If out of these 1000 address and 2000 telephone any data has empty/null sourceID.  I have to process all these records to different filters.

Will it be possible to avoid at tHMap or tXMLMap.

Best Regards

[Anonymous]

Hi Rajender,

    I have not personally tried out about that condition till now. One option is to do an additional aggregation based on sourceID. But doing both loops and aggregation in an XML are costly. So I would recommend to do performance test for extracting data without doing any aggregation at Hmap and filter later scenario and doing aggregation with an additional Hmap condition scenario. Once you get the performance results, you will get better benchmark to choose the right option for your use case.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved