DataPrep 6.4.1 start fails in docker

[cimt_mk]

Hi all,

we are currently setting up a complete Talend environment in docker and we are almost done. However, at the moment I'm struggeling with the DataPrep installation. We are using 6.4.1. In the config file application.properties, I changed the value security.provider to 'tac'. This is the value our current running data prep 6.3.1 instance has. However, when I start dataprep I get the following error:

 Please find the error as text here: https://pastebin.com/d5EKrjrL

This is my application.properties file: https://pastebin.com/3RXL6dKT

The default value of security.provider was oauth, I guess it is for the use of Talend Identity and Access Management. However, I would like to use the normal TAC authentification.

Thanks!

Best regards!

[coheigeartaigh]

This should help with the second error you are seeing - Did you try deleting the "idp" and "oidc" databases after changing the "oidc.host" value to "docker-dev-52.cgn.company.de"? The "passive requestor value" warning is when "oidc.host" does not match the hostname, however it appears that you have "oidc.host" configured correctly. After changing "oidc.host" it's necessary to delete the databases though as per the comment in the configuration file. Could you try doing this and see if step "2" works?

Colm.

[cimt_mk]

Thanks, that actually helped. Of course, I read that comment but I expected that due to the fact that I'm using Docker, with each try I have a clean database anyway. This is basically true. But I forgot that during the build-process of the docker image, I start IAM once. So I already have a "bad" database in my initial image. I have fixed that and the mentioned error is gone!

However, this led me two the next error:

IAM Log:

2018-02-27 12:25:43.068 - INFO [http-nio-9080-exec-7] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 2
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://docker-dev-52.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=IharjG
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[JSESSIONID=6A038137ECA54D9D6345DB3638967D03; TDPSESSION=node0hdk5nkc4l9f1lfblfvceiuzq0.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:25:43.069 - WARN [http-nio-9080-exec-7] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended
27-Feb-2018 12:26:32.164 INFO [http-nio-9080-exec-1] org.apache.cxf.fediz.tomcat8.FederationAuthenticator.authenticate No valid principal found in existing session. Redirecting to IDP
2018-02-27 12:26:51.708 - INFO [http-nio-9080-exec-7] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 3
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://docker-dev-52.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=7q69pP
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[TDPSESSION=node01ledn86cvzzrs11x2l6bgvc6b11.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:26:51.709 - WARN [http-nio-9080-exec-7] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended

I have also created a Data Stewardship container. The error I get there is a bit different:

IAM Log:

2018-02-27 12:34:11.230 - INFO [http-nio-9080-exec-5] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 4
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?consent=none&client_id=tl6K6ac7tSE-LQ&redirect_uri=http://docker-dev-53.cgn.company.de:8080/login&response_type=code&scope=openid%20refreshToken&state=9pAItA
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[JSESSIONID=6A038137ECA54D9D6345DB3638967D03; TDPSESSION=node0hdk5nkc4l9f1lfblfvceiuzq0.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:34:11.231 - WARN [http-nio-9080-exec-5] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended
2018-02-27 12:34:11.237 - WARN [http-nio-9080-exec-5] o.t.i.o.a.BadRequestExceptionMapper      : An OAuth2 BadRequestException occured: HTTP 400 Bad Request
javax.ws.rs.BadRequestException: HTTP 400 Bad Request
        at org.apache.cxf.jaxrs.utils.SpecExceptions.toBadRequestException(SpecExceptions.java:84)
        at org.apache.cxf.jaxrs.utils.ExceptionUtils.toBadRequestException(ExceptionUtils.java:121)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:134)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:122)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:116)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.validateRedirectUri(RedirectionBasedGrantService.java:459)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.startAuthorization(RedirectionBasedGrantService.java:136)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.authorize(RedirectionBasedGrantService.java:96)
        at org.apache.cxf.rs.security.oauth2.services.AuthorizationService.authorize(AuthorizationService.java:58)
        [....] cut off 

Any ideas ? Thanks for the help so far!

[coheigeartaigh]

In terms of the TDS error, the redirect_uri from the log snippet is:

redirect_uri=http://docker-dev-53.cgn.company.de:8080/login

Does this match what is configured in clients/tds-client.json? e.g.:

"redirect_uris" : [ "http://my-machine:19999/login", "http://localhost:19999/login", "http://127.0.0.1:19999/login" ],

 See the following documentation for more information:

https://help.talend.com/reader/vuI_X~V6unFjTgNxRMPcLw/5q~UoTnzcjlhNT0CB1~RJQ

[cimt_mk]

I'm working so long on that setup that I start being disregardful ;-). This was actually the problem. After that, I got some other issues, but these were caused by a missing portmapping (8090) or wrong TAC credentials. Now, it seems to almost work. I can add campaigns or data models. Only when I get to "Semantic Types", I get the following issue:

Spoiler (Highlight to read)

To see the whole post, download it here
OriginalPost.pdf

[cimt_mk]

Hi all,

good news: I got (almost) everything running! Most of the issues were caused by invalid or missing config. The redirecting issue for example can be fixed by changing the IP address in the fediz config file.

bad news: After 6.4.1 was running, we received our license for 6.5.1. The update itself was pretty easy and took only ~30 minutes (thanks to Docker) but I do have a problem with Talend Data Prep now. After logging in, I receive the following message: 

The log message from TDP:

Spoiler (Highlight to read)

To see the whole post, download it here
OriginalPost.pdf

[asharma1]

What is your issuer attribute configuration in this file

• Open the <installation_path>/iam/apache-tomcat/conf/fediz_config.xml file

  <issuer>http://<iam_urlort>/idp/federation</issuer> ?

[cimt_mk]

Hi,

the value is:

<issuer>http://docker-dev-54.cgn.company.de:9080/idp/federation</issuer>

[asharma1]

That looks good. So after 6.5.1 upgrade, TDS is working fine for you, you only have issues with TDP, correct?

Was any password changed for the user you are using to log in to TDP or for TAC security administrator user? Since I see 401 Unauthorized errors or any Role changed for the user you are using to use TDP ?

[cimt_mk]

Yes, correct. After the 6.5.1 upgrade TDP doesn't work, TDS works. Since all Talend components get automatically configured during the creation of the Docker images, nothing should have changed. Only the Talend installation files are different of course (6.5.1 instead of 6.4.1). The TAC database (MySQL) has been migrated as advised (opened with TAC 6.5.1 and used the "migration task"). We didn't change the users.

We use a user called "dpadmin@company.com" which has all privileges (Secruity Admin, Admin etc.). We use this user in a call config files.

Since the TDS connection works, the user itself can't be the reason, right? You are right that the error is 401 Unauthorized, but later in the TDP logs I see also the following error:

Caused by: org.springframework.security.oauth2.common.exceptions.InvalidTokenException: Invalid id_token (iss claim) unexpected issuer : http://docker-dev-54.cgn.company.de:9080/oidc

[coheigeartaigh]

What is the value in TDP 'config/application.properties' for:

* iam.ip

* security.oidc.client.expectedIssuer

The latter needs to match the issuer in the token created by Talend IAM, for your case:

http://docker-dev-54.cgn.company.de:9080/oidc