Modifying Dynamic column metadata to remove illegal column characters

Highlighted
Five Stars

Modifying Dynamic column metadata to remove illegal column characters

Hi,

I want to read dynamic structures from a CSV file into Azure Table Storage to allow for a schemaless design. I am using Talend Cloud Big Data Platform, developing a Standard Batch job.

 

I get the following error:

 

Exception in component tAzureStorageOutputTable_1 (job_Test_ADLS)
org.apache.avro.SchemaParseException: Illegal character in: Prosjektnavn%

 

There are several column names in the CSV containing characters not allowed by the Azure Tables schema definition, such as %, (, ) etc. I want to create a code to alter the column names in the Dynamic column so they comply with the standard.

 

Is there any way to change the column names in the Dynamic column with Java? I would like to avoid changing the column header of the raw input files.

 

Best regards,

Trond


Accepted Solutions
Employee

Re: Modifying Dynamic column metadata to remove illegal column characters

Hi Trond,

 

Yes, it can be done by creating a java routine to change the column names for the dynamic schema. This routine can be called on tMap to make the necessary change.

 

Here is a sample routine to take care of removing special characters like % , $ etc.

 

public static Dynamic modifyDynamicColumns(Dynamic columnSet){
int n=columnSet.getColumnCount();


for (int i=0;i<n;i++){
String name=columnSet.getColumnMetadata(i).getName().replace("%", "").replace("$", "");
columnSet.getColumnMetadata(i).setName(name);
}


return columnSet;

}

 

Hope this helps!

 

Tej Singh

 

 


All Replies
Employee

Re: Modifying Dynamic column metadata to remove illegal column characters

Hi Trond,

 

Yes, it can be done by creating a java routine to change the column names for the dynamic schema. This routine can be called on tMap to make the necessary change.

 

Here is a sample routine to take care of removing special characters like % , $ etc.

 

public static Dynamic modifyDynamicColumns(Dynamic columnSet){
int n=columnSet.getColumnCount();


for (int i=0;i<n;i++){
String name=columnSet.getColumnMetadata(i).getName().replace("%", "").replace("$", "");
columnSet.getColumnMetadata(i).setName(name);
}


return columnSet;

}

 

Hope this helps!

 

Tej Singh

 

 

Five Stars

Re: Modifying Dynamic column metadata to remove illegal column characters

Hi Tej!

 

Thank you very much for a quick and precise reply - I added your code to a tJavaRow component and it works like a charm!

 

Would you happen to know where I can find more java documentation on available methods on the Dynamic data type, such as setName(name) and getColumnMetadata(int)? I could not find any documentation, and I am a java noob...

 

Trond

Employee

Re: Modifying Dynamic column metadata to remove illegal column characters

Hi,

 

There is no specific documentation on the functions I think. In the java code when we use the object and press "Ctrl + Space" with a "dot", it gives the list of functions available for the object. That is what I generally use to get the list of available functions.

 

Thanks!

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Self-service Talend Migration: Moving from On-Premises to the Cloud

Move from On-Premises to the Cloud by following the advice of experts

Read Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read