Load schema dynamically from a file/table

Seven Stars

Load schema dynamically from a file/table

Hi,

 

I was reading this blog

https://www.talend.com/blog/2016/03/11/how-to-operationalize-meta-data-in-talend-with-dynamic-schema... that talks about loading a schema at runtime. This blog looks incomplete and I want to know how to use the dynamically read schema 

 

Does anybody know how to do it or have any other example of loading schema dynamically?

 

My requirement is of reading an input delimited file (CSV) and load its data into a database table.

We want to make it generic so that any file can be read and loaded into a table.

The schema of the file will be defined in another CSV file or database table.

 

I am using the Enterprise Edition, so I know I can use the Dynamic datatype but I have heard that with dynamic datatype the column's datatype are not correctly identified. So would like to know if i can load a schema programatically into a job.


Accepted Solutions
Moderator

Re: Load schema dynamically from a file/table

So you would connect the tSetDynamicSchema to the tMSSqlInput component using an OnComponentOk. Once connected change the schema of the tMSSql Component to take dynamic type and tables name to use the global variable from tMSSqltableList.


All Replies
Moderator

Re: Load schema dynamically from a file/table

Since you have the enterprise version, here is a great example of how to load in a dynamic schema.

Seven Stars

Re: Load schema dynamically from a file/table

Thanks Eric for sharing your blog and the video.

But I am not sure if the solution will work for my requirement.

My requirement is as follows:

The input file will be a CSV file with 'n' columns.

The schema to be read will be defined in a DB table.

The job needs to read the schema metadata from the table.

Then read the input file based on the schema read.

And then output the data read into a DB table.

Moderator

Re: Load schema dynamically from a file/table

you could also do something similar to this where you can use the ttablelist to specify your table and then input it and output it to your target. This will be dynamic due to the ttablelist component 

 

helping with dynamic schema.JPG

Six Stars

Re: Load schema dynamically from a file/table

Please elaborate... Componentwise
Seven Stars

Re: Load schema dynamically from a file/table

The incoming data that the job has to read is from a CSV file and not a table. The data has to be written (output) to a database table. So using tTableList component will not work.

Seven Stars

Re: Load schema dynamically from a file/table

Hi @ehigaki,

 

Can you please elaborate on how I can set the dynamic schema set in a tSetDynamicSchema to a tMSSqlInput component?

 

Thanks,

Ketan

Moderator

Re: Load schema dynamically from a file/table

So you would connect the tSetDynamicSchema to the tMSSqlInput component using an OnComponentOk. Once connected change the schema of the tMSSql Component to take dynamic type and tables name to use the global variable from tMSSqltableList.