topic Re: DataPrep 6.4.1 start fails in docker in Data Quality

DataPrep 6.4.1 start fails in docker

cimt_mk — Sat, 16 Nov 2024 08:40:04 GMT

Hi all,

we are currently setting up a complete Talend environment in docker and we are almost done. However, at the moment I'm struggeling with the DataPrep installation. We are using 6.4.1. In the config file application.properties, I changed the value security.provider to 'tac'. This is the value our current running data prep 6.3.1 instance has. However, when I start dataprep I get the following error:

Please find the error as text here: https://pastebin.com/d5EKrjrL

This is my application.properties file: https://pastebin.com/3RXL6dKT

The default value of security.provider was oauth, I guess it is for the use of Talend Identity and Access Management. However, I would like to use the normal TAC authentification.

Thanks!

Best regards!

Re: DataPrep 6.4.1 start fails in docker

Anonymous — Wed, 21 Feb 2018 10:50:43 GMT

Hi,

As of 6.4.1, Talend IAM is a mandatory piece of the architecture, Data Preparation cannot communicate directly to TAC anymore. So you cannot change the security provider to point to TAC directly. Note that Data Prep users are still managed in TAC's UI. See the following doc page for more details: https://help.talend.com/reader/svfONkQzuB1~g6TYZbfc3Q/ij5BHTinTeKT7wkmiwlrRw & https://help.talend.com/reader/R2jkkk~taTa9YFFcADNN6Q/7GobU5QLUzsZZGXT0q8pDg

Regards,

Gwendal

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Wed, 21 Feb 2018 11:00:14 GMT

Hi Gwendal,

thank you for your response. Well, then we need to install IAM. I have already set up a container for IAM and it's running (I can access it) but I have a very hard time configuring it correctly. The Talend installation guide for IAM is very poor and give no instructions on how to properly configure IAM.

Can you help us with that? Our TAC is connected to LDAP for user authentification. But what are the next steps for configuring IAM? I have entered the IAM URL in TAC but I don't know how to proceed.

Thank you very much!

Re: DataPrep 6.4.1 start fails in docker

Anonymous — Wed, 21 Feb 2018 11:09:11 GMT

Hi,

Have you checked the manual installation procedure? See https://help.talend.com/reader/vuI_X~V6unFjTgNxRMPcLw/wC1xyZN9kSCRCfruACWGCw for IAM and https://help.talend.com/reader/lGdL5d6BacS8BER99BzxIg/_3gzw9IaDWvohZ2KeG62bQ for Prep.

If you still see gaps, please point them out so that we can improve the documentation.

Thank you,

Gwendal

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Wed, 21 Feb 2018 11:20:23 GMT

Hi,

yes, I have checked this documentation. However, the bigger picture is missing for me as I don't exactly know what IAM is doing within the Talend environment.

As I said, IAM is actually already running. But I don't know what I need to configure in IAM, does it need to be connected to TAC? Do I need to configure something within IAM?

I know that the connection between DataPrep and IAM is described in this documentation. But I'm missing the step how to connect IAM to TAC / configure IAM itself.

Edit:

Hm, for some reason my post wasn't saved. Well, here we go again 🙂 :

To see the whole post, download it here
OriginalPost.pdf

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Fri, 23 Feb 2018 19:00:24 GMT

Hi all,

I made some progress. Quick recap:

All components (DataPrep, TAC, IAM etc.) running in individual containers but on the same docker host. I have two issues now:

1. After opening the website of DataPrep, I get redirected to IAM to log in. However, it tries to redirect to a page on "localhost"like "http://localhost:9080/idp/federation?....". When I manually change the localhost to the hostname of my docker server, the redirect works and I can see the Talend IAM login page. When I enter the login credentials, I'll get the following error:

To see the whole post, download it here
OriginalPost.pdf

Re: DataPrep 6.4.1 start fails in docker

Anonymous — Fri, 23 Feb 2018 20:30:10 GMT

Adding @Donn's reply sent by private message:

To see the whole post, download it here
OriginalPost.pdf

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Sat, 24 Feb 2018 09:26:49 GMT

@gvaznunes Funny, now all posts appeared . Feel free to delete the duplicate ones . Btw. I will reach out to you next week with some feedback to the documentation. Now that I know that the feedback actually is getting heard, I'm more than glad to help improving the docs!

I really appreciate the direct communication with the team here, this is a big pro of Talend!

Re: DataPrep 6.4.1 start fails in docker

Anonymous — Sat, 24 Feb 2018 10:23:24 GMT

Wow, that is awkward. Let's keep all the duplicated posts, it could help the Community team understand what happened.

And yes, we truly do listen - there is nothing more valuable than direct user feedback! So thanks again for your help

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Mon, 26 Feb 2018 22:33:45 GMT

I just finished the Data-Stewardship docker container and experience the exact same issue as mentioned above:

1. The redirect for login goes to "localhost" instead of the IAM URL (docker-dev-52.cgn.company.de)

2. When I fix the redirect manually I get to the login page, try to login and get the error in IAM:

 The endpointAddress value of http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize does not match any of the passive requestor values

Now I'm really stuck, can't proceed with our installation without DataPrep and DataStewardship . Appreciate any help!

Re: DataPrep 6.4.1 start fails in docker

coheigeartaigh — Tue, 27 Feb 2018 10:39:55 GMT

This should help with the second error you are seeing - Did you try deleting the "idp" and "oidc" databases after changing the "oidc.host" value to "docker-dev-52.cgn.company.de"? The "passive requestor value" warning is when "oidc.host" does not match the hostname, however it appears that you have "oidc.host" configured correctly. After changing "oidc.host" it's necessary to delete the databases though as per the comment in the configuration file. Could you try doing this and see if step "2" works?

Colm.

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Tue, 27 Feb 2018 12:37:05 GMT

Thanks, that actually helped. Of course, I read that comment but I expected that due to the fact that I'm using Docker, with each try I have a clean database anyway. This is basically true. But I forgot that during the build-process of the docker image, I start IAM once. So I already have a "bad" database in my initial image. I have fixed that and the mentioned error is gone!

However, this led me two the next error:

IAM Log:

2018-02-27 12:25:43.068 - INFO [http-nio-9080-exec-7] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 2
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://docker-dev-52.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=IharjG
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[JSESSIONID=6A038137ECA54D9D6345DB3638967D03; TDPSESSION=node0hdk5nkc4l9f1lfblfvceiuzq0.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:25:43.069 - WARN [http-nio-9080-exec-7] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended
27-Feb-2018 12:26:32.164 INFO [http-nio-9080-exec-1] org.apache.cxf.fediz.tomcat8.FederationAuthenticator.authenticate No valid principal found in existing session. Redirecting to IDP
2018-02-27 12:26:51.708 - INFO [http-nio-9080-exec-7] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 3
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://docker-dev-52.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=7q69pP
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[TDPSESSION=node01ledn86cvzzrs11x2l6bgvc6b11.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:26:51.709 - WARN [http-nio-9080-exec-7] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended

I have also created a Data Stewardship container. The error I get there is a bit different:

IAM Log:

2018-02-27 12:34:11.230 - INFO [http-nio-9080-exec-5] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 4
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?consent=none&client_id=tl6K6ac7tSE-LQ&redirect_uri=http://docker-dev-53.cgn.company.de:8080/login&response_type=code&scope=openid%20refreshToken&state=9pAItA
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[JSESSIONID=6A038137ECA54D9D6345DB3638967D03; TDPSESSION=node0hdk5nkc4l9f1lfblfvceiuzq0.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:34:11.231 - WARN [http-nio-9080-exec-5] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended
2018-02-27 12:34:11.237 - WARN [http-nio-9080-exec-5] o.t.i.o.a.BadRequestExceptionMapper      : An OAuth2 BadRequestException occured: HTTP 400 Bad Request
javax.ws.rs.BadRequestException: HTTP 400 Bad Request
        at org.apache.cxf.jaxrs.utils.SpecExceptions.toBadRequestException(SpecExceptions.java:84)
        at org.apache.cxf.jaxrs.utils.ExceptionUtils.toBadRequestException(ExceptionUtils.java:121)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:134)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:122)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:116)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.validateRedirectUri(RedirectionBasedGrantService.java:459)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.startAuthorization(RedirectionBasedGrantService.java:136)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.authorize(RedirectionBasedGrantService.java:96)
        at org.apache.cxf.rs.security.oauth2.services.AuthorizationService.authorize(AuthorizationService.java:58)
        [....] cut off

Any ideas ? Thanks for the help so far!

Re: DataPrep 6.4.1 start fails in docker

coheigeartaigh — Tue, 27 Feb 2018 15:12:51 GMT

In terms of the TDS error, the redirect_uri from the log snippet is:

redirect_uri=http://docker-dev-53.cgn.company.de:8080/login

Does this match what is configured in clients/tds-client.json? e.g.:

"redirect_uris" : [ "http://my-machine:19999/login", "http://localhost:19999/login", "http://127.0.0.1:19999/login" ],

See the following documentation for more information:

https://help.talend.com/reader/vuI_X~V6unFjTgNxRMPcLw/5q~UoTnzcjlhNT0CB1~RJQ

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Tue, 27 Feb 2018 16:51:31 GMT

I'm working so long on that setup that I start being disregardful ;-). This was actually the problem. After that, I got some other issues, but these were caused by a missing portmapping (8090) or wrong TAC credentials. Now, it seems to almost work. I can add campaigns or data models. Only when I get to "Semantic Types", I get the following issue:

Spoiler (Highlight to read)

To see the whole post, download it here
OriginalPost.pdf

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Mon, 12 Mar 2018 21:03:00 GMT

Hi all,

good news: I got (almost) everything running! Most of the issues were caused by invalid or missing config. The redirecting issue for example can be fixed by changing the IP address in the fediz config file.

bad news: After 6.4.1 was running, we received our license for 6.5.1. The update itself was pretty easy and took only ~30 minutes (thanks to Docker) but I do have a problem with Talend Data Prep now. After logging in, I receive the following message:

The log message from TDP:

Spoiler (Highlight to read)

To see the whole post, download it here
OriginalPost.pdf

Re: DataPrep 6.4.1 start fails in docker

asharma1 — Wed, 14 Mar 2018 12:44:11 GMT

What is your issuer attribute configuration in this file

Open the <installation_path>/iam/apache-tomcat/conf/fediz_config.xml file
<issuer>http://<iam_urlort>/idp/federation</issuer> ?

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Wed, 14 Mar 2018 12:57:36 GMT

Hi,

the value is:

<issuer>http://docker-dev-54.cgn.company.de:9080/idp/federation</issuer>

Re: DataPrep 6.4.1 start fails in docker

asharma1 — Wed, 14 Mar 2018 13:02:07 GMT

That looks good. So after 6.5.1 upgrade, TDS is working fine for you, you only have issues with TDP, correct?

Was any password changed for the user you are using to log in to TDP or for TAC security administrator user? Since I see 401 Unauthorized errors or any Role changed for the user you are using to use TDP ?

Re: DataPrep 6.4.1 start fails in docker

cimt_mk — Wed, 14 Mar 2018 13:22:43 GMT

Yes, correct. After the 6.5.1 upgrade TDP doesn't work, TDS works. Since all Talend components get automatically configured during the creation of the Docker images, nothing should have changed. Only the Talend installation files are different of course (6.5.1 instead of 6.4.1). The TAC database (MySQL) has been migrated as advised (opened with TAC 6.5.1 and used the "migration task"). We didn't change the users.

We use a user called "dpadmin@company.com" which has all privileges (Secruity Admin, Admin etc.). We use this user in a call config files.

Since the TDS connection works, the user itself can't be the reason, right? You are right that the error is 401 Unauthorized, but later in the TDP logs I see also the following error:

Caused by: org.springframework.security.oauth2.common.exceptions.InvalidTokenException: Invalid id_token (iss claim) unexpected issuer : http://docker-dev-54.cgn.company.de:9080/oidc

Re: DataPrep 6.4.1 start fails in docker

coheigeartaigh — Wed, 14 Mar 2018 15:51:04 GMT

What is the value in TDP 'config/application.properties' for:

* iam.ip

* security.oidc.client.expectedIssuer

The latter needs to match the issuer in the token created by Talend IAM, for your case:

http://docker-dev-54.cgn.company.de:9080/oidc