Talend Cloud Databricks Error- modification_time

TCommunity1615436108 — Sat, 16 Nov 2024 00:27:23 GMT

Hi,

I started using Talend Cloud very recently and the use case I am trying is to read data from Delta Lake using Databricks Spark Configuration. I created a Spark Batch job and used my Databricks workspace and cluster in spark configuration. When I execute the job, it is failing immediately with the following error. I tried every runtime version, but it always immediately fails. I couldn't find much conversation on "modification_time" error in Talend. Did anyone else face the same issue?

org.talend.bigdata.launcher.utils.BigDataLauncherException: Error getting content of remote folder '/FileStore/talend/'

at org.talend.bigdata.launcher.fs.DatabricksFileSystem.list(DatabricksFileSystem.java:197)

at org.talend.bigdata.launcher.fs.DatabricksFileSystem.list(DatabricksFileSystem.java:142)

at org.talend.bigdata.launcher.fs.DatabricksFileSystem.updateDatabricksJarList(DatabricksFileSystem.java:565)

at org.talend.bigdata.launcher.databricks.DatabricksJob.uploadJars(DatabricksJob.java:153)

at org.talend.bigdata.launcher.databricks.DatabricksJob.executeJob(DatabricksJob.java:386)

at edap_rbdf.test_0_1.test.runClientJob(test.java:676)

at edap_rbdf.test_0_1.test.runJobInTOS(test.java:480)

at edap_rbdf.test_0_1.test.main(test.java:342)

Caused by: org.talend.bigdata.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "modification_time" (class org.talend.bigdata.launcher.fs.DatabricksFileSystem$FileInfo), not marked as ignorable (3 known properties: "is_dir", "file_size", "path"])

at [Source: (ByteArrayInputStream); line: 1, column: 131] (through reference chain: org.talend.bigdata.launcher.fs.DatabricksFileSystem$FileInfoList["files"]->java.util.ArrayList[0]->org.talend.bigdata.launcher.fs.DatabricksFileSystem$FileInfo["modification_time"])

at org.talend.bigdata.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)

at org.talend.bigdata.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:840)

at org.talend.bigdata.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1206)

at org.talend.bigdata.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1592)

at org.talend.bigdata.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1570)

at org.talend.bigdata.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:294)

at org.talend.bigdata.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)

Re: Talend Cloud Databricks Error- modification_time

Anonymous — Tue, 16 Mar 2021 07:30:07 GMT

@Talend Community , I have seen some related issues open on our bugtracker, have you raised a ticket on Talend Support Portal? If not, please do it and check with our support team if a patch is available.

Regards

Shong

topic Talend Cloud Databricks Error- modification_time in Talend Studio

Talend Cloud Databricks Error- modification_time

Re: Talend Cloud Databricks Error- modification_time