AutoML FAILED_PRECONDITION Error with ScriptEval() Function

[shansen]

This is more of a rhetorical question at this point.   I just wanted to share the answer.  When running the ScriptEval() function in a data load script, I was getting a FAILED_PRECONDITION error on a specific row for each data set, whereas the ScriptEvalEx() function could process the values just fine.

[shansen · Accepted solution]

The problem was that I was using the FARM_FINGERPRINT() function in BigQuery to generate values that I was using as my Association Field.

This is problematic, as FARM_FINGERPRINT() typically generates a large number that Qlik can’t store except as text, but sometimes it has values that are small enough for Qlik to store as numbers, giving the ScriptEval function the wrong data type for matching it up again.  Even when using the Text() function to ensure the FARM_FINGERPRINT field is a string, it sometimes has problems.  So, I found that the best practice is to create a simple RowNo() AS row_num as the data is read-in, and use that as the Association Field in the deployment connection.  Something like:

[PredictionInputs]:

// First 3000 //Load only the first N rows.

LOAD

    num183

    ,num60

    ,category_text

    ,num90

    ,num_total

    ,sk

    ,RowNo() AS row_num

FROM [lib://Application Automation:DataFiles/errinv_prediction_inputs_2024-07-09.csv]

(txt, utf8, embedded labels, delimiter is ',', msq);

// The 5 lines below generate real-time probabilites using the PredictionInputs table fields, and

// join to the PredictionInputs table (above) using the id field.

left join

LOAD

       row_num

    ,event_type_positive AS the_probability

EXTENSION endpoints.ScriptEval('{"RequestType":"endpoint", "endpoint":{"connectionname":"Application Automation:Error Investigation_v1-LGBMC (Machine Learning)"}}', PredictionInputs);

For those who are interested, I am going to include some files so you can reproduce the FAILED_PRECONDITION error, and see how to fix it.  To reproduce the error:

1. Create a new AutoML experiment.
2. Select "errinv_training_dataset_2024-07-09.csv" as the training dataset (Use … > File format settings > Field names:Embedded field names.)
3. Select event_type as the target, and all the others as features.
4. Run the experiment.
5. Select LightGBM Classification algorithm (at least that is the one that was selected last time).
6. Deploy, Deploy. --> Creates "Error Investigation_v1-LGBMC" deployment.
7. Upload (don't analyze) the file "errinv_prediction_inputs_2024-07-09.csv" to the space you are using.
7. Add new > New analytics app > Error Investigation - ML Data Load.qvf > Create
8. In data load editor > Create new connection > Qlik AutoML > ML deployment: Error Investigation_v1-LGBMC (Machine Learning) > Association Field: "sk" > Name: "Error Investigation_v1-LGBMC (Machine Learning)" > Create.
9. Load data.

If the Text() function is removed, the App will throw a FAILED_PRECONDITION error on one the first row where the ScriptEval function cannot match up the data using the sk field. The best way to fix it is to use the RowNo() method described above.