topic Re: Modifying Dynamic column metadata to remove illegal column characters in Talend Studio

Modifying Dynamic column metadata to remove illegal column characters

Anonymous — Sat, 16 Nov 2024 05:00:44 GMT

Hi,

I want to read dynamic structures from a CSV file into Azure Table Storage to allow for a schemaless design. I am using Talend Cloud Big Data Platform, developing a Standard Batch job.

I get the following error:

Exception in component tAzureStorageOutputTable_1 (job_Test_ADLS)
org.apache.avro.SchemaParseException: Illegal character in: Prosjektnavn%

There are several column names in the CSV containing characters not allowed by the Azure Tables schema definition, such as %, (, ) etc. I want to create a code to alter the column names in the Dynamic column so they comply with the standard.

Is there any way to change the column names in the Dynamic column with Java? I would like to avoid changing the column header of the raw input files.

Best regards,

Trond

Re: Modifying Dynamic column metadata to remove illegal column characters

Anonymous — Mon, 12 Aug 2019 04:07:32 GMT

Hi Trond,

Yes, it can be done by creating a java routine to change the column names for the dynamic schema. This routine can be called on tMap to make the necessary change.

Here is a sample routine to take care of removing special characters like % , $ etc.

public static Dynamic modifyDynamicColumns(Dynamic columnSet){
int n=columnSet.getColumnCount();

for (int i=0;i<n;i++){
String name=columnSet.getColumnMetadata(i).getName().replace("%", "").replace("$", "");
columnSet.getColumnMetadata(i).setName(name);
}

return columnSet;

}

Hope this helps!

Tej Singh

Re: Modifying Dynamic column metadata to remove illegal column characters

Anonymous — Mon, 12 Aug 2019 09:16:51 GMT

Hi Tej!

Thank you very much for a quick and precise reply - I added your code to a tJavaRow component and it works like a charm!

Would you happen to know where I can find more java documentation on available methods on the Dynamic data type, such as setName(name) and getColumnMetadata(int)? I could not find any documentation, and I am a java noob...

Trond

Re: Modifying Dynamic column metadata to remove illegal column characters

Anonymous — Tue, 13 Aug 2019 14:41:09 GMT

Hi,

There is no specific documentation on the functions I think. In the java code when we use the object and press "Ctrl + Space" with a "dot", it gives the list of functions available for the object. That is what I generally use to get the list of available functions.

Thanks!

Re: Modifying Dynamic column metadata to remove illegal column characters

vikramk — Mon, 07 Dec 2020 11:19:02 GMT

Hi @TejPratap Singh ,

I have similar requirement, when i use the above code in routine I am getting error as the methods getColumnCount(), getColumnMetadata() are not available hence create methods. How to fix this. Please help.