One Star

Retrieve from Salesforce is missing records

I have a job which uses a tSalesforceInput component to retrieve Salesforce Task records. The job is run daily, and retrieves records where the lastModifiedDate > specified_Date. The output flows to a tMap component and then to a tMSSqlOutput component. The job runs correctly and data flows to the MS SQL table.
While validating a report, I have noticed that there are some records which are not retrieved from Salesforce, and it seems cannot be retrieved. For example, I am missing some task data for a specific contact in Salesforce. When I view the information in Salesforce and compare the results to what I receive using the Talend job, there are records missing.
I am using the option for manual input of the SOQL query. To eliminate possible mapping issues, I select just these columns in the query below and write the results to a tFileOutputDelimited component.
"SELECT Id, AccountId FROM Task where WhoId = '003i000000Q7DPQAA3' "

When I run this job, only 6 out of 7 of the Salesforce task records are retrieved.
As a further test, I have run the same query in Cloudingo Studio, a tool to query Salesforce; 7 out of 7 records are received.
Question: Is there a better way to retrieve the data from Salesforce (daily changes), where I retrieve all the records?

Re: Retrieve from Salesforce is missing records

Is there any data truncation issue on console? What the rejected record look like? Are your records missing randomly?
Best regards
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Retrieve from Salesforce is missing records

Hi Sabrina,
There is no indication of data truncation. I am not sure how to detect if there is a rejected record. There is no indication in the console. When I use Debug Run, I see 6 records process, but the 7th is skipped. The tSalesforceInput does not seem to have an option for rejected rows.
When I view the query and results in Cloudingo Studio, I notice that the field for the row that is not being imported has a much larger size (12849 characters). The schema for this tSalesforceInput component defines this description field as a string with a length of 32000. Note that the query that I am using for this diagnostic testing only includes the Id and AccountId, and does not include the field.
You asked whether the records were being rejected randomly. I will check further into missing records to see if I can find any pattern. However, there are many cases where the record counts are not correct.