Hello folks,
Need help with using a qliksense variable in Spark SQL query.
I have made connection using Simba Spark and its working fine. But could not figure out what i am doing wrong here.
Basically, here i want to fetch data for all the bucket_id in Data table. I can use loop and fetch one by one or i can pass a list using where in statement since spark is querying cassandra and i have constraints with queries. Given query below, would be great if someone can help me with this.
Data:
LOAD bucket_id
resident OD;
LIB CONNECT TO 'Simba Spark (qlik-sense_administrator)';
let noRows = NoOfRows('Data')-1;
for i=0 to $(noRows)
let vVar=FieldValue('bucket_id',$(i));
:
LOAD *, $(vVar);
SQL SELECT bucket_id
FROM SPARK.database1.x
where bucket_id='$(vVar)';
next i
Here, in given above code, i am getting error as :
The following error occurred:
SQL##f - SqlState: S1000, ErrorCode: 35, ErrorMsg: [Simba][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 764.0 failed 4 times, most recent failure: Lost task 0.3 in stage 764.0 (TID 94867, ip-172-31-30-155.ap-southeast-1.compute.internal): java.io.IOException: Exception during execution of SELECT "bucket_id" FROM "database1"."x" WHERE "bucket_id" = ? ALLOW FILTERING: Key may not be empty at com.datastax.spark.connector.rdd.Cas
The error occurred here:
SQL SELECT bucket_id FROM SPARK.database1.x where bucket_id=''
