tSqlRow - SQL Spark Context.

[Anonymous]

Hi,

I want to eliminate duplicate row using tSqlRow. I wrote SQL Hive Context query but want to convert into SQL spark context. Please find below hive query.

"Select B.* from (select row_number() over(partition by id order by acc_id desc, top_id desc)  as  row_numb, A.* from row1 A) as B where row_numb=1"

Can I use row_number using SQL Spark context. I can't use tUniqueRow component.

Thanks.

[Anonymous]

Hello,

if row_number() is a valid spark sql query, you can use it in your job.

Best regards

Sabrina