Test case in Spark Jobs doesn't work with a context variable in tCollectAndCheck

Talend Version       6.x

Summary

 
Additional Versions  
Product Big Data
Component Components
Problem Description

In a Spark Test Case, if tCollectAndCheck uses a context variable (for example, context.reference_file1), then the assertion fails even if the content of the referenced file contains the correct values.

 

It seems Talend uses the value of the context variable itself (the file name) instead of the file content. Here is an extract of execution output:

tCollectAndCheck_1: the reference data does not match the computed results
New elements: true
Missing: true
====================
==== REFERENCE =====
====================
C:\Temp\reference_file.txt
====================
===== RESULTS ======
====================
1;ABC
2;XYZ
====================
= MISSING BY BATCH =
====================
C:\Temp\reference_file.txt
====================
Problem root cause

Despite the name, the variable reference_file1 isn't the path to the reference file, but its content. This context variable value is the one you set in the Test Cases tab, not in the Contexts tab. Also, with Spark, test cases can be tested only from the Test Cases tab, not from the Run tab.

Solution or Workaround
  1. To browse to your reference file, navigate to the Test Cases tab, then under NameOfTheTestCase > Default, select the context reference_file1.
  2. Click File Browse. The content will appear in the text area below; you can edit it as necessary.
  3. Right-click the name of the test case, then select Run TestCase.

    sparkTestCase.jpg

JIRA ticket number  
Version history
Revision #:
2 of 2
Last update:
‎03-09-2018 06:27 PM
Updated by:
 
Contributors