Logging Qlik DB2-to-BigQuery ingestion details into BigQuery table

[sharwesh]

Hi,
I’m ingesting data from DB2 to BigQuery using Qlik. Is there a way to store or log the ingestion details into a BigQuery table? If yes, how can I set it up?

[Andrea_Bertazzo]

Hi Sharwesh,
Thanks for the question:
I think you can find an answer here:
Solved: Export data from qlik to BigQuery - Qlik Community - 1834598

To summarize:

You can modify your Qlik load script to write ingestion metadata (e.g., timestamp, source table, row count) into a BigQuery table using a stored procedure or insert statement.
With the help of AI, I found this code (please do not trust it 100%)
1. Create a logging table in BigQuery:

SQL

CREATE TABLE qlik.ingestion_log (

  source_table STRING,

  target_table STRING,

  row_count INT64,

  load_time TIMESTAMP

);

2. In Qlik load script, after loading data:

LET vSourceTable = 'db2.customers';

LET vTargetTable = 'bigquery.customers';

LET vRowCount = NoOfRows('customers');

LET vLoadTime = Now();

SQL CALL qlik.log_ingestion (

  '$(vSourceTable)',

  '$(vTargetTable)',

  $(vRowCount),

  '$(vLoadTime)'

);

You’ll need to create a stored procedure in BigQuery (log_ingestion) that inserts these values into your log table. This is explained in the post above.
I hope this helps.

[sharwesh]

Hi,

Thanks for you response.
I use Qlik replicate to ingest. But not aware for qlik sense. The above info is mostly related to qlik sense -> Bigquery.
Could you help with Qlik replicate(table log)-> bigquery.
You can let me know if i am missing something too.