Modifying Dynamic column metadata to remove illegal column characters

[Anonymous]

Hi,

I want to read dynamic structures from a CSV file into Azure Table Storage to allow for a schemaless design. I am using Talend Cloud Big Data Platform, developing a Standard Batch job.

I get the following error:

Exception in component tAzureStorageOutputTable_1 (job_Test_ADLS)
org.apache.avro.SchemaParseException: Illegal character in: Prosjektnavn%

There are several column names in the CSV containing characters not allowed by the Azure Tables schema definition, such as %, (, ) etc. I want to create a code to alter the column names in the Dynamic column so they comply with the standard.

Is there any way to change the column names in the Dynamic column with Java? I would like to avoid changing the column header of the raw input files.

Best regards,

Trond

[Anonymous · Accepted solution]

Hi Trond,

Yes, it can be done by creating a java routine to change the column names for the dynamic schema. This routine can be called on tMap to make the necessary change.

Here is a sample routine to take care of removing special characters like % , $ etc.

public static Dynamic modifyDynamicColumns(Dynamic columnSet){
int n=columnSet.getColumnCount();


for (int i=0;i<n;i++){
String name=columnSet.getColumnMetadata(i).getName().replace("%", "").replace("$", "");
columnSet.getColumnMetadata(i).setName(name);
}


return columnSet;

}

Hope this helps!

Tej Singh

[Anonymous]

Hi Tej!

Thank you very much for a quick and precise reply - I added your code to a tJavaRow component and it works like a charm!

Would you happen to know where I can find more java documentation on available methods on the Dynamic data type, such as setName(name) and getColumnMetadata(int)? I could not find any documentation, and I am a java noob...

Trond

[Anonymous]

Hi,

There is no specific documentation on the functions I think. In the java code when we use the object and press "Ctrl + Space" with a "dot", it gives the list of functions available for the object. That is what I generally use to get the list of available functions.

Thanks!

[vikramk]

Hi @TejPratap Singh​ ,

I have similar requirement, when i use the above code in routine I am getting error as the methods getColumnCount(), getColumnMetadata() are not available hence create methods. How to fix this. Please help.