Dynamic Schema with Hive

Four Stars

Dynamic Schema with Hive

Hi Everyone,

We are faced with reading a huge number of hive tables with different schemas. So we were hoping we can develop a single talend job to read these different tables. We recently found out that big data components do not support dynamic schema/type. Does anybody know of a possible solution or work around towards changing schemas with big data components?

Moderator

Re: Dynamic Schema with Hive

Hello,

Are you in Standard DI job or Bigdata batch job?

Here is a new feature jira issue about "Support Hive Metastore HA".

https://jira.talendforge.org/browse/TBD-4059

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Dynamic Schema with Hive

Hi Sabrina

 

There's an article "How to operationalize meta-data in Talend with Dynamic Schemas" by Robert Griswold which mentions you can use a String for big data components since Dynamic Schema is not supported by Big Data components.

 

The blog however doesn't mention how this can be done, do you know a way?

 

Thanks

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now