Those error message seem to come from GoogleQuery and are rather clear.
@SachinB >> You can try to increase the max file size at Target endpoint as per the need. Reload the table.
- I think itis the other way around. The max file size should be decreased to be under 4GB
@suvbin >> the value that has been set for the "Max file size(MB):" in the BigQuery endpoint 1000 gb (max as per the endpoint).
That's just a silly big value suggestion insufficient reflection on the meaning of needs and feeds.
Those are GigaBYTES. Transferred over the network that's 10,000 GigaBits (roughly) and at 1 Gigabit/sec would take 3 hours just to transfer. With the reported Max of 4GB, I would start with 3500 although 4000 MB probably is fine assuming 4GB is really 4096 MB, not 4000 MB but why live on the edge (unless well tested)
Yes 1 file of 4GB is more efficient than 4000 files of 1 MB. But 1 file of 1000GB is unlikely to be measurably more efficient than 250 files of 4GB.
How much change volume/hour are you roughly dealing with? What is the desirable latency? What is the best case upload speed to Google big query from the Replicate server? At that same 1Gigabit/sec a 4 GB file will take 40 seconds, just to transfer. That file also has to be written by Replicate, Read by the network transfer agent, Received by Google, Read by GOogle. I would guess 2 minutes at best for all of that.
Go to 100GB or more and you will exceed all file cache limit everywhere forcing storage write and reads on both sides and hurting everything else on the server in the process, pushing other files out of the caches. Nah, bad idea to go above say 20 GB in general and here above the reported 4GB.
Hein.
