How to filter multiple row dynamically on the basis of String contents

Six Stars

How to filter multiple row dynamically on the basis of String contents

Hi All,

I have a repository based delimited metadata, used as a schema for mapping a delimited  file using tFileInputRegex component(use this component because using regular expression filter record related to gd_header, header, data, trailer from a input source file). schema is as follows:

account_gd_header

date_gd_header

branch_header

debitDetails_header

userValue_data

userdata_data

name_data

name_trailer

userdata_trailer

So i have schema in which every record type (gd_header, header, data, trailer) along with respective column name ism present. So i have to filter above schema. As i am mapping this schema to tMap for filtering purpose. All the records came in as a separate row value. I am unable to filter the record on the basis of record type. How can i achieve this?

Employee

Re: How to filter multiple row dynamically on the basis of String contents

Hi,

 

    Could you please provide screen shots of your jobs and components so that Talend community members will get more insight about your query?

 

Warm Regards,

 

Nikhil Thampi

Six Stars

Re: How to filter multiple row dynamically on the basis of String contents

Hi NIkhilthampi,

 

Thanks for your response. I am giving details of my job:

 

(1) Sample extract file(extractSample.txt):

 

GH|UAT|10819|20180806|12345
H|GB |0.000|43256.78
R|15439|GEC |StaffAccount|239089.00|CB1|Equity|10095
R|16542|NEC |AccountSave|239213.00|CB2|Fund|10096
T|S706 |-66453829.00
GT|DL 0000002576327845|3053|0.00

 

 

(2) Metadata File describing above text file:

ORD|COL_ALIAS|RECORD|SEQ
1|User Defined Value|gd-header|1
1|extname|gd-header|2
1|extid|gd-header|3
1|date|gd-header|4
1|accesscode|gd-header|5
2|User Defined Value|header|1
2|ext_type|header|2
2|open_bal|header|3
2|close_bal|header|4
3|User Defined Value|record|1
3|user_id|record|2
3|user_type|record|3
3|user_account|record|4
3|accnt_balance|record|5
3|account_type|record|6
3|fund_type|record|7
3|account_id|record|8
4|User Defined Value|trailer|1
4|extract_account|trailer|2
4|extract_balance|trailer|3
5|user Defined Value|gd-trailer|1
5|branch_code|gd-trailer|2

 

where,

ORD : order of particular record type in above extract sample file

COL_ALIAS: name of column used is above extract sample file

RECORD: record type identifier for extract sample file

SEQ: sequence of record

 

Each time this metadata will keep on changing as per the extract sample file. My requirement is how to map this metadata for above sample file to match data with respective COL_ALIAS. Any help in this regard will helpful.

Employee

Re: How to filter multiple row dynamically on the basis of String contents

Hi,

 

     From current description, I am assuming that you would like to change the schema details in tMap dynamically. Unfortunately, you cannot dynamically change the schema structure on the fly in tMap component.

 

      I am afraid, the query is still vague from your current description. Could you please share the current job screen shot you have created, the sample input files and output files so that we can help you better.

 

Warm Regards,

 

Nikhil Thampi

Six Stars

Re: How to filter multiple row dynamically on the basis of String contents

Hi Nikhil,

 

Thanks for your suggestion. see i am not able to process or design a job for my query. I am seeking guidance. As this is the file

 

(1) Sample extract file(extractSample.txt):

 

GH|UAT|10819|20180806|12345
H|GB |0.000|43256.78
R|15439|GEC |StaffAccount|239089.00|CB1|Equity|10095
R|16542|NEC |AccountSave|239213.00|CB2|Fund|10096
T|S706 |-66453829.00
GT|DL 0000002576327845|3053|0.00

 

 

(2) Metadata File describing above text file:

ORD|COL_ALIAS|RECORD|SEQ
1|User Defined Value|gd-header|1
1|extname|gd-header|2
1|extid|gd-header|3
1|date|gd-header|4
1|accesscode|gd-header|5
2|User Defined Value|header|1
2|ext_type|header|2
2|open_bal|header|3
2|close_bal|header|4
3|User Defined Value|record|1
3|user_id|record|2
3|user_type|record|3
3|user_account|record|4
3|accnt_balance|record|5
3|account_type|record|6
3|fund_type|record|7
3|account_id|record|8
4|User Defined Value|trailer|1
4|extract_account|trailer|2
4|extract_balance|trailer|3
5|user Defined Value|gd-trailer|1
5|branch_code|gd-trailer|2

 

my expected output is:

 

 

DATA|RECORD|COL_ALIAS

GH|gd-header|User Defined Value

UAT|gd-header|extname

10819|gd-header|extid

20180806|gd-header|date

12345|gd-header|accesscode

 

like that for every record type.

 

it means, I have to map GH record data form extract sample file with gd-header COL_ALIAS. Looking for a job design suggestion.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog