One Star

can I batch load csv files in Hive?

Hello community,
can I use talend in order to load 1000s of csv files into Hive?
if yes, does it matter if the delimiter is , or ; and whether the text fields are within "" or not?
Can the schema be auto detected and the respective tables automatically created?
Is there the functionality of configuring a hotfolder process (the source system is saving csv files in a folder and at regular intervals it is checked whether new files exist in the hotfolder, if yes, then these files are imported into Hive)?
1 REPLY
Seventeen Stars

Re: can I batch load csv files in Hive?

You can do this for sure but at first you never load files into Hive. You load files into the HDFS and you can use Hive to interpret such files as database files. Hive is not a data store, it is a method to run SQL on files on the HDFS with meta data configured in Hive.