CSV file with random Column position

Highlighted
Six Stars

CSV file with random Column position

Hi,

 

I want to read a CSV file from shared drive and process the columns in the file.

 

Name of the columns in the file are same but position on the column is getting changed.

 

E.g.

File1 has columns as "Username","Model","Drive",

File2 can have "Username","Drive", "Model"

 

I want to read the corresponding column and process the data.

 

Could you please let me know how can i handle this situation.

 

 

Highlighted
Community Manager

Re: CSV file with random Column position

I have written a tutorial on precisely this here....

https://www.rilhia.com/tutorials/dynamic-column-order

Highlighted
Forteen Stars

Re: CSV file with random Column position

Yes, it great tutorial!

and I was happy to read it when it was written

 

the only one problem - it really work with only few columns, than start will be more and more complicated (just because a lot of information on screen and a lot of coding)

 

The more universal solutions:

- pack this logic in java routine which loop over any number of columns. Like parse header and sort columns in all other rows by etalon schema (see p.3)

- convert csv to json, again by routine or by any of command line tools, then parse JSON and not worry about order (I use this)

- and final - the best news :-). All what explained in p.1 - already do the great component from Jan Lolling -  tFileInputTextFlat on one of project I extract  with this component  - more than hundreds columns for KPI from source file with 2000+ columns , work perfect!

 

 

-----------

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog