Unable to create hive external table with avro.schema.url

Four Stars

Unable to create hive external table with avro.schema.url

Hello Experts,

 

I'm trying to create Hive external table using talend data fabric 7.1.1 but although I gave avro.schema.url the table creation is throwing an error "at least one column must be specified for the table". I have .avsc file kept at the avro.schema.url location.

I'm using thivecreatetable component with create external table box checked, built in schema is empty 'cuz I want to retrieve it from .avsc file located at avro.schema.url. 

 

Table Properties                        value

"avro.schema.url"                    "hdfs://path/filename.avsc"

 

Could somebody help please? 

Moderator

Re: Unable to create hive external table with avro.schema.url

Hello,

We currently don't have a way in the Studio to automate getting the Avro Schema from a file, especially on HDFS.

To do it manually, please have a look at avro-tools (see http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ for a good tutorial). 

There's a "getschema" command, but it requires the avro file to be on the local filesystem.  This is probably the best way to do it for one or two tables.

Please refer to this new feature jira issue:https://jira.talendforge.org/browse/TBD-5870

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Unable to create hive external table with avro.schema.url

Hello,

Even if I get schema retrieved from avro file, my question is how do I create external table based on the .avsc file, which is not working with tHiveCreateTable component? what is the best way to create external table based on .avsc file? 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now