I have a requirement to use globalMap in Spark bigdata suit.
I understand in 6.x version this is not supported , Are there any alternatives. I need to populate the globalmap with some static values and reference them in the tmap transformation.
there are about 200 entries for the map. i can get that job working in DI but not in Bigdata enc.
My flor is
file --> tjavaflex(populate globalmap) --> in another subjob access it in tmap.
In fact we have no access for "globalMap" in Spark Batch mode.(because of the different implementation way on Spark Batch compared to DI)
The reason is that it's difficult to have a synchronous "global" variable in distributed mode and in addition the globalMap is not totally serializable by default.
Here are some articles about context in spark job.
Hope it will shed some light on your requirement.
I ran into that article too but my requirement dosent change.
i need to pass a hashmap to the spark jobs if not globalmap then some other way .
I tried using hashmap as context variable but couldn't cast it back to hashmap from string from a context value in sparkjob.
my flow was
Inputfile --> tjava(populate hashmap and add to context varibale the hashmap) --> pass context to spark job --> tmap access hashmap (error: cant cast string to hashmap)
Would you mind posting your current job setting screenshots on forum? Which will be helpful for us to get more information.
Please mask your sensitive data.
my use case was to load a static business file which is of the forum k,v
The problem i cannot use it as a map lookup because the different rows attributes, pass different attributes some static some non static to find a value . So to calculate x if i pass (col1) from main flow , to calculate y i pass col2. etc. so i cannot use any set of same columns as part of map key.
I resolved this by writing procedure to read this file and storing it as a string buffer then parse this string in loop with record separator and try to match and return the matched result.