TIS : append data into parquet file

Highlighted
Six Stars

TIS : append data into parquet file

Hi,
I am using TIS Version: 6.2.1 .

I want to read data from Teradata and write it into HDFS (HDP v2.4, scoop > 1.4.6) with parquet file format
My first thought was to use tScoopImport component 
The problem is that the version of sqoop deployed in my cluster is 1.4.6 which is  buged and it doesn't support custom sql.
So I turned to parquet file :
I  created a spark job and  I used tFileOutputParquet that works juste fine but he didn't support appending data
to the same file even when I do set column partition ! 
So my question is :  How can I append data in a parquet file written in HDFS ?
Thanks you a lot dor your help.
Moderator

Re: TIS : append data into parquet file

Hi,
 Per TalendHelpCenter:Which big data formats are supported, parquet is not suppored on tSqoopImport
Could you please take a look at this jira issue:https://jira.talendforge.org/browse/TBD-3392 to see if the workaround is Ok with you?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now