Six Stars

tSqlRow - SQL Spark Context.

Hi,

 

I want to eliminate duplicate row using tSqlRow. I wrote SQL Hive Context query but want to convert into SQL spark context. Please find below hive query.

 

"Select B.* from (select row_number() over(partition by id order by acc_id desc, top_id desc)  as  row_numb, A.* from row1 A) as B where row_numb=1"

 

Can I use row_number using SQL Spark context. I can't use tUniqueRow component.

 

Thanks.

1 REPLY
Moderator

Re: tSqlRow - SQL Spark Context.

Hello,

if row_number() is a valid spark sql query, you can use it in your job.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.