I want to eliminate duplicate row using tSqlRow. I wrote SQL Hive Context query but want to convert into SQL spark context. Please find below hive query.
"Select B.* from (select row_number() over(partition by id order by acc_id desc, top_id desc) as row_numb, A.* from row1 A) as B where row_numb=1"
Can I use row_number using SQL Spark context. I can't use tUniqueRow component.
if row_number() is a valid spark sql query, you can use it in your job.