Dynamic Schema with Hive

Four Stars

Dynamic Schema with Hive

Hi Everyone,

We are faced with reading a huge number of hive tables with different schemas. So we were hoping we can develop a single talend job to read these different tables. We recently found out that big data components do not support dynamic schema/type. Does anybody know of a possible solution or work around towards changing schemas with big data components?

Moderator

Re: Dynamic Schema with Hive

Hello,

Are you in Standard DI job or Bigdata batch job?

Here is a new feature jira issue about "Support Hive Metastore HA".

https://jira.talendforge.org/browse/TBD-4059

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Dynamic Schema with Hive

Hi Sabrina

 

There's an article "How to operationalize meta-data in Talend with Dynamic Schemas" by Robert Griswold which mentions you can use a String for big data components since Dynamic Schema is not supported by Big Data components.

 

The blog however doesn't mention how this can be done, do you know a way?

 

Thanks

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads