topic data loading from PostgreSQL through source connection failed in Catalog and Lineage

data loading from PostgreSQL through source connection failed

hemant3 — Tue, 07 Dec 2021 22:35:04 GMT

Hi All,

I am facing an issue while loading data from PostgreSQL via source connection when i load the logs

below are the logs from QDC

2020-06-17 17:10:45.270: Starting job: cate.categories.20200617171023 2020-06-17 17:10:45.272: Started loading into dock. 2020-06-17 17:10:45.287: Watching for process completion/termination. 2020-06-17 17:10:45.456: Task DockHandLoader failed: IOException: Mkdirs failed to create file:/datacatalyst/loadingdock/cate/categories/20200617171023 (exists=false, cwd=file:/usr/local/qdc/apache-tomcat-7.0.94/webapps/qdc/WEB-INF/classes)

thanks

Re: data loading from PostgreSQL through source connection failed

DaveHowland — Wed, 17 Jun 2020 19:57:20 GMT

@hemant3 ,

Can you confirm that the QDC Service was started as the correct user. Any chance you started it as 'root' or another non-qdc user? Also .. did you change the directory where QDC stores data (in the core_env.properties file)?

If you're running as the QDC user , but changed the directory you're writing to '/datacatalyst/loadingdock/' .. make sure you've done the right chmod to give that qdc user the right, recursively, on that directory. Like 'chown -R qdc:qdc /datacatalyst/loadingdock/ ' or something like it.

If you do that and try it again, I would expect it to work, but if it doesn't let us know what error you're getting.

Thanks!

David

Re: data loading from PostgreSQL through source connection failed

hemant3 — Thu, 18 Jun 2020 11:16:06 GMT

Hi David,

i have checked with access rights of qdc user and they are fine i had checked the same by creating a directory through command line where it was successfully created on standalone system.

and all services are up and running with qdc user

still i am facing same error

below is the log

2020-06-18 16:26:57.423: Starting job: customer_demographic.customer_customer_demo.20200618162638 2020-06-18 16:26:57.427: Started loading into dock. 2020-06-18 16:26:57.460: Watching for process completion/termination. 2020-06-18 16:26:57.645: Task DockHandLoader failed: IOException: Mkdirs failed to create file:/datacatalyst/loadingdock/customer_demographic/customer_customer_demo/20200618162638 (exists=false, cwd=file:/usr/local/qdc/apache-tomcat-7.0.94/webapps/qdc/WEB-INF/classes)

Re: data loading from PostgreSQL through source connection failed

DaveHowland — Thu, 18 Jun 2020 12:02:49 GMT

@hemant3

From your core_env.properties: can you provide the value for your 'podium.base=' ?

Also ..as important : are you able to onboard other sources / entities beyond this? Is it just this one?
And what is the 'base directory' set to for this source?