topic Retrieve from Salesforce is missing records in Talend Studio

Retrieve from Salesforce is missing records

Anonymous — Sat, 16 Nov 2024 11:37:23 GMT

I have a job which uses a tSalesforceInput component to retrieve Salesforce Task records. The job is run daily, and retrieves records where the lastModifiedDate > specified_Date. The output flows to a tMap component and then to a tMSSqlOutput component. The job runs correctly and data flows to the MS SQL table.
While validating a report, I have noticed that there are some records which are not retrieved from Salesforce, and it seems cannot be retrieved. For example, I am missing some task data for a specific contact in Salesforce. When I view the information in Salesforce and compare the results to what I receive using the Talend job, there are records missing.
I am using the option for manual input of the SOQL query. To eliminate possible mapping issues, I select just these columns in the query below and write the results to a tFileOutputDelimited component.

"SELECT Id, AccountId FROM Task where WhoId = '003i000000Q7DPQAA3' "

When I run this job, only 6 out of 7 of the Salesforce task records are retrieved.
As a further test, I have run the same query in Cloudingo Studio, a tool to query Salesforce; 7 out of 7 records are received.
Question: Is there a better way to retrieve the data from Salesforce (daily changes), where I retrieve all the records?
Thanks,
Tom

Re: Retrieve from Salesforce is missing records

Anonymous — Fri, 06 Jun 2014 04:33:05 GMT

Hi,
Is there any data truncation issue on console? What the rejected record look like? Are your records missing randomly?
Best regards
Sabrina

Re: Retrieve from Salesforce is missing records

Anonymous — Fri, 06 Jun 2014 14:15:24 GMT

Hi Sabrina,
There is no indication of data truncation. I am not sure how to detect if there is a rejected record. There is no indication in the console. When I use Debug Run, I see 6 records process, but the 7th is skipped. The tSalesforceInput does not seem to have an option for rejected rows.
When I view the query and results in Cloudingo Studio, I notice that the field for the row that is not being imported has a much larger size (12849 characters). The schema for this tSalesforceInput component defines this description field as a string with a length of 32000. Note that the query that I am using for this diagnostic testing only includes the Id and AccountId, and does not include the field.
You asked whether the records were being rejected randomly. I will check further into missing records to see if I can find any pattern. However, there are many cases where the record counts are not correct.
Thanks,
Tom