A little bit of the background, we are trying to setup an ODS. That means we are pulling data from Salesforce to a SQL server. It is a daily job performed on TAC.
It was running fine for a while, until an user told us that the data count was not match. So we did a little bit of research. Later on we found out that when we perform a full extract which means pulling 34mil of records. Only 24mil of records were managed to get into our SQL server.
As we investigated a little bit further, we found that the tSalesforceInput is not getting all data from the bulk job. As we compare the Number of lines from tSalesforceInput and the number of records on Bulk Job log. The counts drops from 34mil to 24mil which matches the phenomena above.
We have another further investigation, and found out that there is a field on SF that is free text and can contain up to 130k characters. And if we remove that field from the job, it works properly. So we have the conclusion that when the data exceed a certain amount, tSalesforceInput cannot fetch all the data from bulk API.
So, the question here is, has anyone also experience this before?
