Knowledge

[SwathiPulagam]

Environment

Qlik Replicate 
BigQuery Endpoint

By default, BigQuery will work with gzip compression from 2022.05 and we observed that loading is 3x faster when we are using compression.

Note: For any reason, if we need to set compression off then we have to add a feature flag "CompressedFastLoad-GBQTarget" to the BQ target endpoint and set it to "false"(By default, this feature is ON and consult support if you want to change feature flag).

Here is the log snippet from 2021.11 and 2022.05

From 2021.11 log:

00081280: 2022-05-13T08:30:20:595945 [AT_GLOBAL ]I: Task Server Log - SQL_GBQ (V2021.11.0.391 ******.qliktech.com Microsoft Windows 8 Enterprise Edition (build 9200) 64-bit, Revision:403b59717b9941cf699383d1eff7db7f025b1b23, PID: 80084) started at Fri May 13 08:30:20 2022 (at_logger.c:2687)
.
.
00085996: 2022-05-13T08:31:04:25399 [TARGET_LOAD ]T: Going to load file 'C:\Program Files\Attunity\Replicate\data\tasks\SQL_GBQ\data_files\1\LOAD00000001.csv' (csv_target.c:1629)
00085996: 2022-05-13T08:31:04:25935 [TARGET_LOAD ]T: Load data command. Path: , Exe name: bq, Exe params: --project_id qlik-support-automation --dataset_id dbo load --allow_quoted_newlines=true --format=csv --null_marker="attNULL" attreg_maxtableTest1 "C:\\Program Files\\Attunity\\Replicate\\data\\tasks\\SQL_GBQ\\data_files\\1\\LOAD00000001.csv" (provider_syntax_manager.c:3598)
00085996: 2022-05-13T08:31:04:25935 [TARGET_LOAD ]T: Going to run load statement, '/bq' (csv_target.c:1138)

From 2022.05 log:

00076708: 2022-05-13T08:27:46:649395 [AT_GLOBAL ]I: Task Server Log - SQL_GBQ (V2022.5.0.291 ******.qliktech.com Microsoft Windows 8 Enterprise Edition (build 9200) 64-bit, Revision:a82fb96beb16d0018595df2a09cbd912edbc4f05, PID: 77024) started at Fri May 13 08:27:46 2022 (at_logger.c:2693)
.
.
00026136: 2022-05-13T08:28:24:310782 [TARGET_LOAD ]T: Going to load file 'C:\Program Files\Attunity\Replicate\data\tasks\SQL_GBQ\data_files\1\LOAD00000001.csv' (csv_target.c:1689)
00026136: 2022-05-13T08:28:24:317089 [TARGET_LOAD ]T: File C:\Program Files\Attunity\Replicate\data\tasks\SQL_GBQ\data_files\1\LOAD00000001.csv compressed to gzip (csv_target.c:1533)
00026136: 2022-05-13T08:28:24:317089 [TARGET_LOAD ]T: Load data command. Path: , Exe name: bq, Exe params: --project_id qlik-support-automation --dataset_id dbo load --allow_quoted_newlines=true --format=csv --null_marker="attNULL" attreg_maxtableTest1 "C:\\Program Files\\Attunity\\Replicate\\data\\tasks\\SQL_GBQ\\data_files\\1\\LOAD00000001.csv.gz" (provider_syntax_manager.c:3584)
00026136: 2022-05-13T08:28:24:317089 [TARGET_LOAD ]T: Going to run load statement, '/bq' (csv_target.c:1143)