Ok, so I'm trying out Talend as it seems like the right option for what I need (getting some local DB from multiple locations and of multiple types back to a singular data warehouse). So I am just messing about to try to understand it. I had a local CSV file on my desktop and I have been trying to just insert it into BigQuery. Excluding the mime.type warning I am a bit stuck on the 6 column thing. I eyeballed the txt that was generated and there are a few lines with 201804 but they aren't at the top (halfway through and the full value is cut off). I am not sure why this is happening and I'm at the stage of just randomly changing CSV options thinking that might help. Sorry if this isn't the right place to get help but I am very newbish with this and I'm just trying to figure out if it's the right thing or not to use for my use case.
[statistics] connecting to socket on port 3747
[statistics] connected
[WARN ]: org.jets3t.service.utils.Mimetypes - Unable to find 'mime.types' file in classpath
Exception in component tBigQueryOutput_1_tBQBE (TestCSVtoBigQuery)
java.lang.RuntimeException: Job failed: BigQueryError{reason=invalid, location=gs://shared/biquery_bulk.txt, message=Error while reading data, error message: CSV table references column position 6, but line starting at position:20804 contains only 6 columns.}
at local_project.testcsvtobigquery_0_1.TestCSVtoBigQuery.tFileInputDelimited_1Process(TestCSVtoBigQuery.java:2259)
at local_project.testcsvtobigquery_0_1.TestCSVtoBigQuery.runJobInTOS(TestCSVtoBigQuery.java:2609)
at local_project.testcsvtobigquery_0_1.TestCSVtoBigQuery.main(TestCSVtoBigQuery.java:2459)
[statistics] disconnected
