Load CSV data to Google BigQuery Failed

[LY112233]

Hi all,

I have encountered below error when trying to uploading a csv file with size larger than 4GB as below:

   Exception in component tBigQueryOutput_2_tBQBE

   java.lang.RuntimeException: Job failed: BigQueryError{reason=invalid, location=gs://xx, message=Error while reading data, error message: The options set for reading CSV prevent BigQuery from splitting files to read in parallel, and at least one of the files is larger than the maximum allowed size when files cannot be split. Size is: 77188694016. Max allowed size is: 4294967296.}

I make some searches on the google and found this in the stack overflow (https://stackoverflow.com/questions/44784757/error-loading-large-csv-into-google-bigquery)

where it mentions two possible cases:

   1. CSV file was compressed

   2. There is a quoting character mismatch in one of the fields, which makes it look like very long string in that field, also making file not splittable.

The csv file is uncompressed. Hence, I believe is the "

options set for reading CSV" causing the error. Can someone help me on this? Thanks in advance.

[Anonymous]

Hello,

The mismatched quoting in CSV file? Did you try to check the CSV option in the tFileInputDelimited component to output as CSV file?

Best regards

Sabrina

[LY112233]

Hi Sabrina,

Do you mean that checking the below option? I checked it before and the same error is still happening.

 

Thanks and regards,

Low

[Anonymous]

Hello,

Any example Input Data in .csv-file?

If your source file is enclosed by double quotes, you need to configure it like this.

Best regards

Sabrina

[LY112233]

Hi,

I am sorry that I couldnt provide you the Input Data due to confidential reasons but all the columns within the csv file are either string contains one alphanumeric word or numeric value without comma or other special characters or symbols. So I believe it is not enclosed by double quotes.

Thank You.

Best Regards,

Low

[febiariw]

Hi,

I ever found the similar error that mentioned : "java.lang.RuntimeException: Job failed: BigQueryError{reason=invalid, location=null, message=Error while reading data, error message: CSV processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0}"

even the source data that i used was gsheet file, the error message said that CSV processing something....

I'm not sure that the error identified was because bigquery table doesn't appropriate. but when I delete that table and make some change talend configuration in tBigQueryOutput and checked the option to "create the table if it doesn't exist", then run the job.

after doing the step above, no error found, it works...