How to process multiple input files with different columns in talend

Five Stars

How to process multiple input files with different columns in talend

Hi Experts,

 

I am new to talend and want to understand how to process multiple files in same job with different column names.

 

For ex:

text1:

account

identity

customer

 

text2:

acctn

id

cus

 

I have thousands file like given above and I have to implement same business logic for every file only challenge is column names are different in every file. So please let me know how dynamically we can process these files in talend.

 

thanks in advance Smiley Happy 

Sixteen Stars TRF
Sixteen Stars

Re: How to process multiple input files with different columns in talend

Column names don't matter as soon as the number of column and their signification are the same for all the files.


TRF
Seven Stars

Re: How to process multiple input files with different columns in talend

Hi @abhityagi,

 

It will be better if you can tell , what exactly you want. Expalin better so that we can help you better.

 

Regards

Five Stars

Re: How to process multiple input files with different columns in talend

ok thanks for your reply. Let me give you one example.

 

I have thousands of file all the files have different names and different column names. I want to apply business logic such as if any column is null then assigned 'Y' value to new variable or assigned 'N'. I don 't want to create again and again schema for any new file or for column names. I want to create a job which can read any file with any column name convention and also create new variable as per their assigned value.

 

file1:

col:

account

name

gender

 

file2:

col:

acctn 

cust

sex

 

Thanks,

Abhishek

Moderator

Re: How to process multiple input files with different columns in talend

Hello,

The Dynamic schema feature in talend allows you to design schema with an unknown column structure (unknown name and number of columns).

Please refer to this KB article about:https://community.talend.com/t5/Design-and-Development/How-to-process-changing-data-structure/ta-p/2...

Let us know if it is OK with you.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog