Qlik Community

Knowledge

Search or browse our knowledge base to find answers to your questions ranging from account questions to troubleshooting error messages. The content is curated and updated by our global Support team

Qlik Replicate 2022.05 BigQuery target endpoint Feature: BigQuery load performance improved using compression

cancel
Showing results for 
Search instead for 
Did you mean: 
SwathiPulagam
Support
Support

Qlik Replicate 2022.05 BigQuery target endpoint Feature: BigQuery load performance improved using compression

Environment

Qlik Replicate 
BigQuery Endpoint

 

By default, BigQuery will work with gzip compression from 2022.05 and we observed that loading is 3x faster when we are using compression.

Note: For any reason, if we need to set compression off then we have to add a feature flag "CompressedFastLoad-GBQTarget" to the BQ target endpoint and set it to "false"(By default, this feature is ON and consult support if you want to change feature flag).

Here is the log snippet from 2021.11 and 2022.05

From 2021.11 log:

00081280: 2022-05-13T08:30:20:595945 [AT_GLOBAL ]I: Task Server Log - SQL_GBQ (V2021.11.0.391 ******.qliktech.com Microsoft Windows 8 Enterprise Edition (build 9200) 64-bit, Revision:403b59717b9941cf699383d1eff7db7f025b1b23, PID: 80084) started at Fri May 13 08:30:20 2022 (at_logger.c:2687)
.
.
00085996: 2022-05-13T08:31:04:25399 [TARGET_LOAD ]T: Going to load file 'C:\Program Files\Attunity\Replicate\data\tasks\SQL_GBQ\data_files\1\LOAD00000001.csv' (csv_target.c:1629)
00085996: 2022-05-13T08:31:04:25935 [TARGET_LOAD ]T: Load data command. Path: , Exe name: bq, Exe params: --project_id qlik-support-automation --dataset_id dbo load --allow_quoted_newlines=true --format=csv --null_marker="attNULL" attreg_maxtableTest1 "C:\\Program Files\\Attunity\\Replicate\\data\\tasks\\SQL_GBQ\\data_files\\1\\LOAD00000001.csv" (provider_syntax_manager.c:3598)
00085996: 2022-05-13T08:31:04:25935 [TARGET_LOAD ]T: Going to run load statement, '/bq' (csv_target.c:1138)


From 2022.05 log:

00076708: 2022-05-13T08:27:46:649395 [AT_GLOBAL ]I: Task Server Log - SQL_GBQ (V2022.5.0.291 ******.qliktech.com Microsoft Windows 8 Enterprise Edition (build 9200) 64-bit, Revision:a82fb96beb16d0018595df2a09cbd912edbc4f05, PID: 77024) started at Fri May 13 08:27:46 2022 (at_logger.c:2693)
.
.
00026136: 2022-05-13T08:28:24:310782 [TARGET_LOAD ]T: Going to load file 'C:\Program Files\Attunity\Replicate\data\tasks\SQL_GBQ\data_files\1\LOAD00000001.csv' (csv_target.c:1689)
00026136: 2022-05-13T08:28:24:317089 [TARGET_LOAD ]T: File C:\Program Files\Attunity\Replicate\data\tasks\SQL_GBQ\data_files\1\LOAD00000001.csv compressed to gzip (csv_target.c:1533)
00026136: 2022-05-13T08:28:24:317089 [TARGET_LOAD ]T: Load data command. Path: , Exe name: bq, Exe params: --project_id qlik-support-automation --dataset_id dbo load --allow_quoted_newlines=true --format=csv --null_marker="attNULL" attreg_maxtableTest1 "C:\\Program Files\\Attunity\\Replicate\\data\\tasks\\SQL_GBQ\\data_files\\1\\LOAD00000001.csv.gz" (provider_syntax_manager.c:3584)
00026136: 2022-05-13T08:28:24:317089 [TARGET_LOAD ]T: Going to run load statement, '/bq' (csv_target.c:1143)

Labels (1)
Version history
Last update:
2 weeks ago
Updated by: