One Star djl
One Star

Registering Hive UDF's (User Defined Functions)

I have successfully set up and used a Hive input connector to connect to Hadoop - running in Amazon EMR (by using a JDBCInput connector and specifying all the required Hive/Hadoop libs).
The standard Hive queries work fine.
My only issue is that I cannot figure out how to register and use my UDF's as part of a job flow.
ie
add jar s3://path/libs/my_lib.jar;
create temporary function my_func as 'mylib.MyFunction';
select my_func(col_a) from ...

My UDF queries successfully run in the SQLBuilderDialog but when I attempt to run them as part of a job, the job returns almost instantly without error - with output being an empty file with the just the column headers.
Can someone please point in the correct direction for registering and using UDF's
Im using TOS 4.2.2
4 REPLIES
Community Manager

Re: Registering Hive UDF's (User Defined Functions)

Hi
I am interesting in your job, can you export the job and send it to me?
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Employee

Re: Registering Hive UDF's (User Defined Functions)

try putting the first two line "Add jar" and "create function" in seperate jdbc row calls and then issue the select using tjdbcinput. see if that helps.
KArthik
Five Stars

Re: Registering Hive UDF's (User Defined Functions)

any solution?
I have the same issue when I'm trying to use a function stored in HIVE
How to register the function in hive session? with thiverow first? or another way to do?
In shell: first:
create temporary function findInMsg as 'com.r9.scrub.hive.FindInMessage'"
and after: select findInMsg('local',message) from table r
I'm trying to do the same with thiverow / thiveinput but it doesn't work
thiverow:
create temporary function findInMsg as 'com.scrub.hive.FindInMessage'
> subjob ok
> tinputrow
select
findInMsg('local',message),
from
table r

Any idea?
My error:
Exception in component tHiveInput_2
java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:194)
One Star

Re: Registering Hive UDF's (User Defined Functions)

I know its a very very old thread but was there any solution found on this issue?