How to Eliminate Local dataset (file) in Big Query Output,bulkexec component while using the Talend Cloud Integration

Anonymous — Sat, 16 Nov 2024 08:22:17 GMT

Hi,

We are using Talend cloud Studio to develop the jobs, then we are required to publish them in TIC. As we are using the Google Big Query as our DB (Source & Destination), We are using the tBigqueryInput & tBigqueryOutput component. Both components require local files, as the input component need the token file and output component need local file to keep the data file , before uploading the dataset to cloud storage or insert them into google big query table.

As mentioned earlier, we need to publish them to Talend Cloud and carry out the loads in TIC. Now we need to know, how to eliminate the local file use, is possible to use only the cloud storage, we have access to google cloud storage, where we can store the files.

Using Talend Cloud Studio 6.5 JDBC-ELT (not supporting big query datatype) and JDBC is (slow) (Simba jdbc), As we need to process 500 million records in few tables and also we need a solution to eliminate local file .

As our requirement is to have the ETL, only in Talend cloud platform and use the talend cloud engine. (no remote engine). So need help, to resolve the issue

topic How to Eliminate Local dataset (file) in Big Query Output,bulkexec component while using the Talend Cloud Integration in Talend Studio

How to Eliminate Local dataset (file) in Big Query Output,bulkexec component while using the Talend Cloud Integration