tBigQueryBulkExec importing CSV error: You are loading data without specifying data format.

[jamesasu]

I have tried setting up a simple pipeline in Talend Studio that takes Opportunity records from Salesforce, drops them into a CSV, uploads to Google Cloud Storage and then imports into a BigQuery table. Everything works up until the BigQuery import, at which point I get the following error:

Exception in component tBigQueryBulkExec_1 (Opps_to_BigQuery)
java.lang.RuntimeException: Job failed: BigQueryError{reason=invalid, location=null, message=You are loading data without specifying data format, data will be treated as CSV format by default. If this is not what you mean, please specify data format by --source_format.}
	at sf_opps_to_bigquery.opps_to_bigquery_0_1.Opps_to_BigQuery.tBigQueryBulkExec_1Process(Opps_to_BigQuery.java:481)
	at sf_opps_to_bigquery.opps_to_bigquery_0_1.Opps_to_BigQuery.runJobInTOS(Opps_to_BigQuery.java:983)
	at sf_opps_to_bigquery.opps_to_bigquery_0_1.Opps_to_BigQuery.main(Opps_to_BigQuery.java:783)
[FATAL]: sf_opps_to_bigquery.opps_to_bigquery_0_1.Opps_to_BigQuery - tBigQueryBulkExec_1 Job failed: BigQueryError{reason=invalid, location=null, message=You are loading data without specifying data format, data will be treated as CSV format by default. If this is not what you mean, please specify data format by --source_format.}

Has anyone experienced this or know a way to get BigQuery to read from the repository schema?

[Anonymous]

Does the same thing happen if you untick the "Bulk file already exists in Google Storage" and use that mechanism to send the data?

[jamesasu]

Good thought. I've just tried that and unfortunately, that gives me the same error.

[Anonymous]

Hi,

   Could you please check whether the previous tBigQueryOutputBulk is correctly pointing to the right area inside Google Storage? Especially the file extension.

    There is a solution Template for Salesforce to Google BigQuery in Talend Academy. I would recommend to try the solution with the Template.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[jamesasu]

I was able to get this working in the end by going to the Advanced Settings and checking the "Set the field delimiter" checkbox.