How to transform the dynamic column before inserting it into desired target?

Six Stars

How to transform the dynamic column before inserting it into desired target?

Hi,

 

I'm getting the data from source in dynamic column. Lets say 4 columns comes in it. So, I want to apply some transform over 2 columns then send the data into target/output.

 

And want a dynamic solution for achieving this situation like columns will be created dynamically in target side.

 

Please suggest......

Highlighted
Thirteen Stars

Re: How to transform the dynamic column before inserting it into desired target?

how do you identify columns to work on in dynamics?
if you have 4 columns why don't you get 4 columns?

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Highlighted
Six Stars

Re: How to transform the dynamic column before inserting it into desired target?

Real time situation is like we are getting a file on daily basis and its structure is changing mostly on daily basis.

 

For example, on day-1 we receive a file with 4 columns, on day-2 we receive a file with 6 columns(2 new columns added) similarly on day-3 we receive a file with 3 columns only(3 columns removed from file) so on......

 

So, how I can handle this situation dynamically so that my columns at target(AWS S3) side will get adjusted based on the source file. I don't want to pre-define any schema on both source and target side.

 

Also, can I also transform the columns that are coming in source file before sending them into target?

Highlighted
Thirteen Stars

Re: How to transform the dynamic column before inserting it into desired target?

You can use tJavaFlex to work (using java code) on Dynamic fields.Good luck.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Highlighted
Six Stars

Re: How to transform the dynamic column before inserting it into desired target?

Any help document or material available for using tjavaflex with an example?
Highlighted
Six Stars

Re: How to transform the dynamic column before inserting it into desired target?

Instead of writing java code can i write s python code for transforming the columns ?
Highlighted
Thirteen Stars

Re: How to transform the dynamic column before inserting it into desired target?

no, Talent is a java code generator...

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Highlighted
Thirteen Stars

Re: How to transform the dynamic column before inserting it into desired target?

int ncount=input_row.c.getColumnCount();
DynamicMetadata column0=row1.c.getColumnMetadata(ncount-2);
DynamicMetadata column1=row1.c.getColumnMetadata(ncount-1);
System.out.println("The column name is:"+column0.getName()+" The length of column is: "+column0.getLength()+" The column Type is: "+column0.getType());
System.out.println("The column name is:"+column1.getName()+" The length of column is: "+column1.getLength()+" The column Type is: "+column1.getType());


it's a sample of code but usefull.
Thanks to @shong

Francois Denis

Tag as "solved" for others! Kudos to thanks!

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now