When using Amazon EMR with Glue as storage, the following error can be encountered when performing a Full Load of more than 13 tables:
sqlstate 'HY000', errorcode '500051', message '[Amazon][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code: 10006, SQL state: TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: SemanticException [Error 10006]: Partition not found ... Query: ALTER TABLE `ap_gldb_lf_processed_atlas_use_dev`.`qlik_cmps_status` DROP IF EXIST
Information provided on this defect is given as is at the time of documenting. For up-to-date information, please review the most recent Release Notes with RECOB-2894 for reference
Environment
Qlik Compose Data Lakes - 2021.2.xx
Fix Version
Fixed in 2021.5.0.78 and higher.
Cause
The length of the expression we need for querying the partitions info for the tables included in the task has a limitation for the length of the query, and it's documented here:
https://docs.aws.amazon.com/glue/latest/webapi/API_GetPartitions.html
Jira issue: RECOB-2894
