Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Fetching a very large dataset (> 10 GB) using GoogleBig Query from Qlik OdbcConnectorPackage 6.149 returns:
[Qlik][BigQuery] (100) Error interacting with REST API: Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/troubleshooting-errors
The following configuration is required in the connection for large data sets:
If the driver doesn't show the options, those can be configured in the "Advance Parameters" section
SQLDialect 0 (Legacy SQL)
AllowLargeResults 1 (Enabled)
LargeResultsDataSetID <destination datasetname>
Driver version 6.149 has an option in the UI.
In 6.149 version, when working with a very large dataset, a temporary destination table is needed within GBQ to stage the data before it is made available to an external application. The 6.149, GoogleBig Query connector exposes the parameters to use for a temporary table. The user will have to manually create a dataset in advance in GBQ when using these parameters (i.e. when it is expected to load large amounts of data).
Steps: (the following are suggested steps. Recommend to consult with internal GBQ admin)
Notes:
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
If a data fetch is larger than 10 GB, Google requires a temporary table.
Qlik Google BigQuery Connector: How to setup load from Large Datasets