I am trying to update context variable on run time from flat file. It is a big data spark job.
I am reading csv file where it has context variable and their value. With tJavaRow component, I am assigning context variable value. The same approach is working fine with Standard job but not with Big data job.
Where I am wrong in this approach? How we can assign values to context variables in Big data spark job during run time from flat file?
For BD batch jobs, the Talend recommendation is to use a DI job 'launcher' that loads the "dynamic" context.
So far, there is also no tContextLoad component for Big Data Batch and Spark Streaming.
Watch the recorded webinar!
Accelerate your data lake projects with an agile approach
Create systems and workflow to manage clean data ingestion and data transformation.
Introduction to Talend Open Studio for Data Integration.