Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi,
I started using Talend Cloud very recently and the use case I am trying is to read data from Delta Lake using Databricks Spark Configuration. I created a Spark Batch job and used my Databricks workspace and cluster in spark configuration. When I execute the job, it is failing immediately with the following error. I tried every runtime version, but it always immediately fails. I couldn't find much conversation on "modification_time" error in Talend. Did anyone else face the same issue?
org.talend.bigdata.launcher.utils.BigDataLauncherException: Error getting content of remote folder '/FileStore/talend/'
at org.talend.bigdata.launcher.fs.DatabricksFileSystem.list(DatabricksFileSystem.java:197)
at org.talend.bigdata.launcher.fs.DatabricksFileSystem.list(DatabricksFileSystem.java:142)
at org.talend.bigdata.launcher.fs.DatabricksFileSystem.updateDatabricksJarList(DatabricksFileSystem.java:565)
at org.talend.bigdata.launcher.databricks.DatabricksJob.uploadJars(DatabricksJob.java:153)
at org.talend.bigdata.launcher.databricks.DatabricksJob.executeJob(DatabricksJob.java:386)
at edap_rbdf.test_0_1.test.runClientJob(test.java:676)
at edap_rbdf.test_0_1.test.runJobInTOS(test.java:480)
at edap_rbdf.test_0_1.test.main(test.java:342)
Caused by: org.talend.bigdata.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "modification_time" (class org.talend.bigdata.launcher.fs.DatabricksFileSystem$FileInfo), not marked as ignorable (3 known properties: "is_dir", "file_size", "path"])
at [Source: (ByteArrayInputStream); line: 1, column: 131] (through reference chain: org.talend.bigdata.launcher.fs.DatabricksFileSystem$FileInfoList["files"]->java.util.ArrayList[0]->org.talend.bigdata.launcher.fs.DatabricksFileSystem$FileInfo["modification_time"])
at org.talend.bigdata.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
at org.talend.bigdata.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:840)
at org.talend.bigdata.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1206)
at org.talend.bigdata.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1592)
at org.talend.bigdata.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1570)
at org.talend.bigdata.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:294)
at org.talend.bigdata.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
@Talend Community , I have seen some related issues open on our bugtracker, have you raised a ticket on Talend Support Portal? If not, please do it and check with our support team if a patch is available.
Regards
Shong