One Star

Dynamically generate Talend packages

Guys,
I recently started looking at Talend Open Studio for Big data and I like most of the options available.
But i have questions with Dynamic column type and it is not supported in Big Data components, is there any options available to accomplish this.
Like SSIS(Microsoft ETL tool) can i extend the functionality, for example changing the schema on the fly or creating an api to generate talend packages programmatically.
Thanks for your help.
Anand.

  • Big Data
7 REPLIES
Moderator

Re: Dynamically generate Talend packages

Hi,
Are you looking for "Dynamic Schema" which allows you to design schema with an unknown column structure (unknown name and number of columns)? TalendHelpCenter:How to process changing data structure?.
The dynamic schema functionality is only available in Talend enterprise subscription version not open source.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Dynamically generate Talend packages

Thanks Sabrina.
But Dynamic schema was not supported in Big Data version, in our use cases we want to reuse the packages.
Again is there a way change the schema at run time in Big Data/ regular enterprise version?
Or can we extend Talend API to achieve this?
Thanks,
Anand.
Moderator

Re: Dynamically generate Talend packages

Hi,
I recently started looking at Talend Open Studio for Big data and I like most of the options available.

The dynamic schema functionality is only available in Talend enterprise subscription version.(Talend Enterprise big data.. not open source).
So far, schemas must be defined during design, not run time.
Please have a look at talend KB article https://help.talend.com/pages/viewpage.action?pageId=5671106.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Dynamically generate Talend packages

Thanks for the quick reply.
Yes I am evaluating Talend Big Data Enterprise edition and it does not support Dynamic data type for Big Data components, one example i have tested so far tHiveInput?
Do you have any sample job for Hive Input to File Output or Vice Versa with Dynamic Column Data Type?
Thanks,
Anand.
Moderator

Re: Dynamically generate Talend packages

Hi,
So far, thivexxx components don't support for "dynamic schema".
You can go to /plugins/org.talend.core.tis_x.x.x.rxxxxx.jar.
unzip this jar to a directory, and you will see a file call supportDynamic.txt under resources folder, the file lists all the components support the dynamic schema in the current version.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Dynamically generate Talend packages

Thanks Sabrina.
Moderator

Re: Dynamically generate Talend packages

Hi,
It's welcome to open a feature jira issue of DI project on Talend Bug Tracker.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.