topic Query condition for tSalesforceInput based on Ids in a CSV File in Talend Studio

Query condition for tSalesforceInput based on Ids in a CSV File

Anonymous — Sat, 16 Nov 2024 11:54:34 GMT

Hi,
We are running a talend (With a salesforce connection) job to Loads Task object-records in a .csv file (using tFileOutputDelimited).
We wanted to restrict the number of records queried for an Object based on Ids in a file delimiter (CSV file) . Could someone please help us here, as we are new to talend.

Thanks in advance!

Re: Query condition for tSalesforceInput based on Ids in a CSV File

Anonymous — Tue, 10 Sep 2013 17:42:04 GMT

Can you better explain your issue.
Are you reading or writing to Salesforce?
What exactly are you trying to achieve?

Re: Query condition for tSalesforceInput based on Ids in a CSV File

Anonymous — Wed, 11 Sep 2013 07:24:53 GMT

Hey Thanks for a prompt response!
We are reading and writing both to Salesforce.
We are supposed to load Tasks from an Old Production to a New Production instance and there are about 28 lakhs task records in old org but we need to query only around 2 lakh records. We have the set of IDs to be queried in a CSV file.
We need a query condition which looks like : ""what.type = 'case' AND whoid = null and createddate" AND Id = '' (this is to be taken with a tFileInputDelimited ... right?)
PS: We cannot use tMap and then match the ids because according to our understanding it queries all the 28 lakh records and then matches them; but with that size, its running to heap size issues (Even if we increase the heap size in the .ini config file) and then time-out errors consistently, so there itself is the glitch.
WE WANT TO FILTER THEM WHILE QUERING ONLY using the CSV file , PLEASE SUGGEST IF THERE IS A BETTER WAY !

Re: Query condition for tSalesforceInput based on Ids in a CSV File

Anonymous — Wed, 11 Sep 2013 07:35:58 GMT

In Short,
We thought we can reference the CSV file in the query condition of a tSalesforceInput to query only when the Task object Id = 'Ids in the CSV File', ONLY, we don't really know how to do that.
I hope I am clear now

Re: Query condition for tSalesforceInput based on Ids in a CSV File

Anonymous — Wed, 11 Sep 2013 09:11:23 GMT

Ah...lakh=100000
It's a tricky one as it's a 'cloud' service.
Couldn't you reapply the logic to Salesforce that gave you the CSV files of IDs in the first place?
If not, I'd break things down.
Export all contenders from Salesforce to a CSV file.
Join your two files using tMap.
You can increase HEAP - Run->Advances Settings
tMap supports "Store temp data" which conserves memory; but slows things down
Exporting large Salesforce datasets with Talend is a challenge, especially if you want to restrict in this way.

Re: Query condition for tSalesforceInput based on Ids in a CSV File

alevy — Thu, 12 Sep 2013 15:49:40 GMT

See https://community.talend.com/t5/Design-and-Development/How-to-perform-Dynamic-Lookup/td-p/104177.