I need to define metadata for array of values and need someway to use it in spark sql. Or atleast tmap
i need metadata type for d1...d40
Do you want to define your Array of values under Metadata in repository tree or Contexts value? Could you please give us more information about your requirement?
requirement is simple , you can think of it as i have delimited data coming in but 40 of which are discount %ges for same product on different days.
prodattrib1| prodattrib2|disc1|disc2|disc3| ..........disc40|prodattribx|prodattriby
i want to represent the addtributes disc1 ...disc40 in a vector or array because i will have to do some decimal calculations based on it.
Currently i have 40 attributes for each discount which is really not that friendly especially when i want to loop those values, i am converting them to stringbuffer and looping thru -it very awkward.
in our previous etl tool we could do that so just wondering if i can do with talend.