Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
ThiebaudS
Partner - Creator II
Partner - Creator II

Spark JDBC Driver for Databricks

Hello,

I'm trying to use Compose for Data Lake with Databricks and I'm facing a strange issue with the Spark JDBC Driver.

I've downloaded the driver from the Databricks website and placed it in the C:\Program Files\Attunity\Compose for Data Lakes\java\jdbc folder, as mentioned in the documentation.

Despite this, in Compose when I try to add my storage connection, I receive the following error : 

SYS-E-HTTPFAIL, Failed to add session connection: SYS-E-GNRLERR, Required driver class not found: com.simba.spark.jdbc41.Driver.

I don't understand because I also have the Hive JDBC Driver in the exact same folder and this one is working with Compose.

I've restarted all Attunity services and even the server with no luck.

Am I missing something ?

Thank you very much.

Best regards,

Thiebaud

Labels (1)
1 Solution

Accepted Solutions
TimGarrod
Employee
Employee

Apologies - I was responding to an email I got from the community - which didn't include the screenshots (doh!).

Can you tell me the version of C4DL you are running?  

Also - has C4DL previously been configured to use a remote agent?  (Go to the main Compose screen - Management> Compose Agent Settings).  (This would typically only be the case if you had an EMR / Hadoop / HDI etc Spark project running - but just want to check 🙂 ).

TimGarrod_0-1607444435342.png

Agent settings -   If this is set to "Remote" then you must ensure the Spark driver is located where your agent is running (and then restart the agent via compose-agent.sh stop  and compose-agent.sh start).

TimGarrod_1-1607444607478.png

 

 

View solution in original post

6 Replies
TimGarrod
Employee
Employee

Did you install the SparkJDBC41.jar driver or the 42 driver? Compose 6.6 requires the 41 driver. If this is not available on the databricks download site you may need to contact their support. (It was on their download site, but I believe they may have removed it recently and replaced with 42 driver).
ThiebaudS
Partner - Creator II
Partner - Creator II
Author

Hi @TimGarrod 

I've installed the SparkJDBC41.jar, as you can see on the screenshot in my previous post.

Thanks

TimGarrod
Employee
Employee

Apologies - I was responding to an email I got from the community - which didn't include the screenshots (doh!).

Can you tell me the version of C4DL you are running?  

Also - has C4DL previously been configured to use a remote agent?  (Go to the main Compose screen - Management> Compose Agent Settings).  (This would typically only be the case if you had an EMR / Hadoop / HDI etc Spark project running - but just want to check 🙂 ).

TimGarrod_0-1607444435342.png

Agent settings -   If this is set to "Remote" then you must ensure the Spark driver is located where your agent is running (and then restart the agent via compose-agent.sh stop  and compose-agent.sh start).

TimGarrod_1-1607444607478.png

 

 

ThiebaudS
Partner - Creator II
Partner - Creator II
Author

Hi again @TimGarrod 

Thanks a lot for your help.

I'm using C4DL 6.6.0.940, and yes I had the remote server option checked for the agent settings because I was previously working on a Hadoop cluster...

I've set the agent settings to local and I'm now receiving this new error message:

SYS-E-HTTPFAIL, Failed to add session connection: SYS-E-SQLERR, Sql statement produced the following error: Failed to get connection: jdbc:spark://adb-********.azuredatabricks.net:443;transportMode=http;httpPath=sql/protocolv1/o/********/********-roils75;SSL=1;AuthMech=3;PWD=****** Error message: [Simba][SparkJDBCDriver](500151) Error setting/closing session: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Hostname and certificate CN are mismatched....

Does it tell you something ?

Thanks again !

 

ThiebaudS
Partner - Creator II
Partner - Creator II
Author

Nevermind, I've added "AllowSelfSignedCerts=1" in the "JDBC parameters" field in Compose, and the connection is now working.

Thanks a lot @TimGarrod 

Best regards,

Thiebaud

raghavan
Contributor
Contributor

The JayDeBeApi module allows you to connect from Python code to databases using Java JDBC. It provides a Python DB-API v2. 0 to that database. It works on ordinary Python (cPython) using the JPype Java integration or on Jython to make use of the Java JDBC driver.

typhoon tv