multischema multi level xml data filter Talend data mapper

Highlighted
Seven Stars

multischema multi level xml data filter Talend data mapper

Hi  there

 

I have a multi schema and multi level XML.  File has a tag originSource at starting loop and then each multi schema and Level has a tag SourceID.  I don't want data to load into my tables if it has a source "MySource" and any of SourceID in multi schema and level is empty. But if Source it is "YourSource" and multi schema and level sourceID is null then data should be loaded successfully.

 

Is it possible to filter data using Talend Data Mapper.  I tried write filter condition on validate tag in child node. Then It is removing only that particular data. My expectation is to remove whole hierarchy in such case.  

 

Any other approach to handle it is welcome.

 

Thank you for your time.

 

Best Regards

Employee

Re: multischema multi level xml data filter Talend data mapper

Hi,

 

   If its just empty tags in XML, it will not have any impact in your final data right?

 

   But if you want to remove the empty XML tags also, I am not sure as I have not personally tried it. Lets hear others' opinions also.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Seven Stars

Re: multischema multi level xml data filter Talend data mapper

Hi Nikhil

 

Thanks for your time. Let me put my query in this way.

 

Not the empty tag I want to remove. I don't want whole set of Node. For example.

 

<persons>

<sourceID>

<OriginSource>

  <name>

  <age>

  <address>

  <sourceID>

    <street>

    <state>

    <pin>

 </address>

 <telephone>

   <sourceID>

  <personal>

  <office>

  <fax>

</telephone>

</persons>

 

There can be many person in one xml. If any of the person(X) is not "MySource" and either  Address --> sourceID or telephone --> sourceID is null. Then that person(X) should not be loaded.

 

Best Regards

 

 

 

Employee

Re: multischema multi level xml data filter Talend data mapper

Hi,

 

   Could you please share the screen shot of the job you are trying to do? If you want to remove any empty records, you can try to remove them later using tFilterRow component later in the flow. 

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Seven Stars

Re: multischema multi level xml data filter Talend data mapper

Hi

 

There are 7 tables loaded from tXMLMap. So I am thinking to filter such records in XML itself.

Following is my job design.

 

fileList --> tfileInputRaw --> tHmap -> tXMLMap -> 7 output to DB tables.

 

Best Regards

 

Employee

Re: multischema multi level xml data filter Talend data mapper

Hi Rajender,

 

     I believe you are extracting the records from XML to tabular format before loading data to DB.

 

     You can add a tFilterRow after that component to remove the unwanted data.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Seven Stars

Re: multischema multi level xml data filter Talend data mapper

Hi 

 

Yes, you are right.  

 

Please refer XML example mention in previous post, One person(XYZ) has 1000 address and 2000 telephone. If out of these 1000 address and 2000 telephone any data has empty/null sourceID.  I have to process all these records to different filters.

 

Will it be possible to avoid at tHMap or tXMLMap.

 

Best Regards

 

Employee

Re: multischema multi level xml data filter Talend data mapper

Hi Rajender,

 

    I have not personally tried out about that condition till now. One option is to do an additional aggregation based on sourceID. But doing both loops and aggregation in an XML are costly. So I would recommend to do performance test for extracting data without doing any aggregation at Hmap and filter later scenario and doing aggregation with an additional Hmap condition scenario. Once you get the performance results, you will get better benchmark to choose the right option for your use case.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog