Spark JDBC Driver for Databricks

[ThiebaudS]

Hello,

I'm trying to use Compose for Data Lake with Databricks and I'm facing a strange issue with the Spark JDBC Driver.

I've downloaded the driver from the Databricks website and placed it in the C:\Program Files\Attunity\Compose for Data Lakes\java\jdbc folder, as mentioned in the documentation.

Despite this, in Compose when I try to add my storage connection, I receive the following error : 

SYS-E-HTTPFAIL, Failed to add session connection: SYS-E-GNRLERR, Required driver class not found: com.simba.spark.jdbc41.Driver.

I don't understand because I also have the Hive JDBC Driver in the exact same folder and this one is working with Compose.

I've restarted all Attunity services and even the server with no luck.

Am I missing something ?

Thank you very much.

Best regards,

Thiebaud

[TimGarrod]

Did you install the SparkJDBC41.jar driver or the 42 driver? Compose 6.6 requires the 41 driver. If this is not available on the databricks download site you may need to contact their support. (It was on their download site, but I believe they may have removed it recently and replaced with 42 driver).

[ThiebaudS]

Hi @TimGarrod 

I've installed the SparkJDBC41.jar, as you can see on the screenshot in my previous post.

Thanks

[TimGarrod · Accepted solution]

Apologies - I was responding to an email I got from the community - which didn't include the screenshots (doh!).

Can you tell me the version of C4DL you are running?  

Also - has C4DL previously been configured to use a remote agent?  (Go to the main Compose screen - Management> Compose Agent Settings).  (This would typically only be the case if you had an EMR / Hadoop / HDI etc Spark project running - but just want to check 🙂 ).

Agent settings -   If this is set to "Remote" then you must ensure the Spark driver is located where your agent is running (and then restart the agent via compose-agent.sh stop  and compose-agent.sh start).

[ThiebaudS]

Hi again @TimGarrod 

Thanks a lot for your help.

I'm using C4DL 6.6.0.940, and yes I had the remote server option checked for the agent settings because I was previously working on a Hadoop cluster...

I've set the agent settings to local and I'm now receiving this new error message:

SYS-E-HTTPFAIL, Failed to add session connection: SYS-E-SQLERR, Sql statement produced the following error: Failed to get connection: jdbc:spark://adb-********.azuredatabricks.net:443;transportMode=http;httpPath=sql/protocolv1/o/********/********-roils75;SSL=1;AuthMech=3;PWD=****** Error message: [Simba][SparkJDBCDriver](500151) Error setting/closing session: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Hostname and certificate CN are mismatched....

Does it tell you something ?

Thanks again !

[ThiebaudS]

Nevermind, I've added "AllowSelfSignedCerts=1" in the "JDBC parameters" field in Compose, and the connection is now working.

Thanks a lot @TimGarrod 

Best regards,

Thiebaud

[raghavan]

The JayDeBeApi module allows you to connect from Python code to databases using Java JDBC. It provides a Python DB-API v2. 0 to that database. It works on ordinary Python (cPython) using the JPype Java integration or on Jython to make use of the Java JDBC driver.

typhoon tv