Adding new column dynamically in Talend Free Version

Four Stars

Adding new column dynamically in Talend Free Version

Hi everyone,

 

I am using Talend Open Studio Free version 6.3.0.

I have a requirement to add new columns to the schema dynamically. Example: I am reading a table from a db and one day a client adds one new column in the db-table then I should be able to accommodate the newly added column in the Talend schema dynamically, without changing the code. This schema is propagated to the  output file I am generating. How can we achieve this. Please suggest solutions. Thanks!!

Community Manager

Re: Adding new column dynamically in Talend Free Version

This can't be done. If the input or output schema are changed you will need to modify your job to accommodate this. You *could* use a query stored outside of your job to concatenate all columns of your input database into a comma separated String, read this query in during the job, then use it in your input component. If you had a single column for your input component, this would allow you to kind of achieve this output to a file. I suggest this as if you are outputting to a file, you will have to convert all data types to String anyway. Therefore you can create a single job and only have to modify a query stored externally to the job. This will limit what you can do with the data though. 

Four Stars

Re: Adding new column dynamically in Talend Free Version

Hi rhall_2_0,

Thanks for your response. Actually I need to do some transformations also for every incoming column, eg, left padding zeroes, right padding with spaces, then few columns data to be left aligned and few right aligned. So concatenating all the columns won't help. Is there any other trick? Please suggest.. Thanks!!

Regards.
Highlighted
Community Manager

Re: Adding new column dynamically in Talend Free Version

You can still do this with the method I suggested. It would just take some String manipulation inside the job. However, it seems your transformation rules are not consistent. This leads me to ask, how you expect to implement your rules if you do not know of the columns you will receive? I think you need to tell your client that if they change the schema, they will have to change the job. 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog