I have a job which uses a tSalesforceInput component to retrieve Salesforce Task records. The job is run daily, and retrieves records where the lastModifiedDate >
specified_Date. The output flows to a tMap component and then to a tMSSqlOutput component. The job runs correctly and data flows to the MS SQL table.
While validating a report, I have noticed that there are some records which are not retrieved from Salesforce, and it seems cannot be retrieved. For example, I am missing some task data for a specific contact in Salesforce. When I view the information in Salesforce and compare the results to what I receive using the Talend job, there are records missing.
I am using the option for manual input of the SOQL query. To eliminate possible mapping issues, I select just these columns in the query below and write the results to a tFileOutputDelimited component.
"SELECT Id, AccountId FROM Task where WhoId = '003i000000Q7DPQAA3' "
When I run this job, only 6 out of 7 of the Salesforce task records are retrieved.
As a further test, I have run the same query in Cloudingo Studio, a tool to query Salesforce; 7 out of 7 records are received.
Question: Is there a better way to retrieve the data from Salesforce (daily changes), where I retrieve
all the records?
Thanks,
Tom
Hi Sabrina, There is no indication of data truncation. I am not sure how to detect if there is a rejected record. There is no indication in the console. When I use Debug Run, I see 6 records process, but the 7th is skipped. The tSalesforceInput does not seem to have an option for rejected rows. When I view the query and results in Cloudingo Studio, I notice that the field for the row that is not being imported has a much larger size (12849 characters). The schema for this tSalesforceInput component defines this description field as a string with a length of 32000. Note that the query that I am using for this diagnostic testing only includes the Id and AccountId, and does not include the field. You asked whether the records were being rejected randomly. I will check further into missing records to see if I can find any pattern. However, there are many cases where the record counts are not correct. Thanks, Tom
