tSqlRow - SQL Spark Context.

Six Stars

tSqlRow - SQL Spark Context.



I want to eliminate duplicate row using tSqlRow. I wrote SQL Hive Context query but want to convert into SQL spark context. Please find below hive query.


"Select B.* from (select row_number() over(partition by id order by acc_id desc, top_id desc)  as  row_numb, A.* from row1 A) as B where row_numb=1"


Can I use row_number using SQL Spark context. I can't use tUniqueRow component.




Re: tSqlRow - SQL Spark Context.


if row_number() is a valid spark sql query, you can use it in your job.

Best regards


Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.