tSalesforceInput component fails with the error, 'InvalidBatch : Failed to process query: OPERATION_TOO_LARGE: exceeded 100000 distinct ids'

Problem Description

A tSalesforceInput component using the Bulk or Query mode, fails with the following error:

[FATAL]: oes_data_warehouse.extract_salesforce_emailmessage_0_2.Extract_Salesforce_EmailMessage - tSalesforceInput_1 org.talend.daikon.exception.error.DefaultErrorCode@dc9876b:{failedBatch=[BatchInfo  id='7510l000001lO53AAE'
jobId='7500l000001KRyRAAW'
state='Failed'
stateMessage='InvalidBatch : Failed to process query: OPERATION_TOO_LARGE: exceeded 100000 distinct ids'
createdDate='java.util.GregorianCalendar[time=1528346642000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2018,MONTH=5,WEEK_OF_YEAR=23,WEEK_OF_MONTH=2,DAY_OF_MONTH=7,DAY_OF_YEAR=158,DAY_OF_WEEK=5,DAY_OF_WEEK_IN_MONTH=1,AM_PM=0,HOUR=4,HOUR_OF_DAY=4,MINUTE=44,SECOND=2,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1528346643000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2018,MONTH=5,WEEK_OF_YEAR=23,WEEK_OF_MONTH=2,DAY_OF_MONTH=7,DAY_OF_YEAR=158,DAY_OF_WEEK=5,DAY_OF_WEEK_IN_MONTH=1,AM_PM=0,HOUR=4,HOUR_OF_DAY=4,MINUTE=44,SECOND=3,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]

Adjusting the Batch Size does not help.

 

Root Cause

This is a limitation of Salesforce.

For more information, refer to:

SAP Knowledge Base article, 2270580 - Error: 'HY000 : OPERATION_TOO_LARGE:Exceeded 100000 distinct ids' when refreshing a report...

Salesforce, Developers, Discussion Forum, SOQL Count() query fails with OPERATION_TOO_LARGE. Why?

 

Solution

Salesforce recommends:

Refining the filtering criteria to reduce the amount of data or adding the permission View All Data to the Salesforce.com user.

Version history
Revision #:
9 of 9
Last update:
‎06-14-2019 09:20 PM
Updated by: