Six Stars

DataPrep 6.4.1 start fails in docker

Hi all,

 

we are currently setting up a complete Talend environment in docker and we are almost done. However, at the moment I'm struggeling with the DataPrep installation. We are using 6.4.1. In the config file application.properties, I changed the value security.provider to 'tac'. This is the value our current running data prep 6.3.1 instance has. However, when I start dataprep I get the following error:

dataprep_error.png

 Please find the error as text here: https://pastebin.com/d5EKrjrL

 

This is my application.properties file: https://pastebin.com/3RXL6dKT

 

The default value of security.provider was oauth, I guess it is for the use of Talend Identity and Access Management. However, I would like to use the normal TAC authentification.

 

Thanks!

Best regards!

Tags (2)
20 REPLIES
Employee

Re: DataPrep 6.4.1 start fails in docker

Hi,

 

As of 6.4.1, Talend IAM is a mandatory piece of the architecture, Data Preparation cannot communicate directly to TAC anymore. So you cannot change the security provider to point to TAC directly. Note that Data Prep users are still managed in TAC's UI. See the following doc page for more details: https://help.talend.com/reader/svfONkQzuB1~g6TYZbfc3Q/ij5BHTinTeKT7wkmiwlrRwhttps://help.talend.com/reader/R2jkkk~taTa9YFFcADNN6Q/7GobU5QLUzsZZGXT0q8pDg

 

Regards,

 

Gwendal

Six Stars

Re: DataPrep 6.4.1 start fails in docker

Hi Gwendal,

 

thank you for your response. Well, then we need to install IAM. I have already set up a container for IAM and it's running (I can access it) but I have a very hard time configuring it correctly. The Talend installation guide for IAM is very poor and give no instructions on how to properly configure IAM. 

 

Can you help us with that? Our TAC is connected to LDAP for user authentification. But what are the next steps for configuring IAM? I have entered the IAM URL in TAC but I don't know how to proceed.

 

Thank you very much!

Employee

Re: DataPrep 6.4.1 start fails in docker

Hi,

 

Have you checked the manual installation procedure? See https://help.talend.com/reader/vuI_X~V6unFjTgNxRMPcLw/wC1xyZN9kSCRCfruACWGCw for IAM and https://help.talend.com/reader/lGdL5d6BacS8BER99BzxIg/_3gzw9IaDWvohZ2KeG62bQ for Prep.

 

If you still see gaps, please point them out so that we can improve the documentation.

 

Thank you,

 

Gwendal

Six Stars

Re: DataPrep 6.4.1 start fails in docker

Hi,

 

yes, I have checked this documentation. However, the bigger picture is missing for me as I don't exactly know what IAM is doing within the Talend environment. 

 

As I said, IAM is actually already running. But I don't know what I need to configure in IAM, does it need to be connected to TAC? Do I need to configure something within IAM? 

 

I know that the connection between DataPrep and IAM is described in this documentation. But I'm missing the step how to connect IAM to TAC / configure IAM itself.

 

Edit:

 

 

Hm, for some reason my post wasn't saved. Well, here we go again :-) :

 

I managed to install and (almost) configure everything properly. Remember, all Talend components are in independent containers, though on the same docker server / host. At the moment, I have two issues:

 

1. When I go to the DataPrep Website, I get redirected to IAM in order to log in. However, I get redirected to "localhost", like that:

 

http://localhost:9080/idp/federation?wa=wsignin1.0&..........

 

 

When I change localhost to docker-dev-52.cgn.company.com, I get redirected to IAM. After logging in with the data prep user I have created in TAC, it leads me to issue 2:

 

2018-02-23 19_50_31-Error.png

 

The IAM log shows the following:

 

 

2018-02-23 19:41:20.878 -ERROR [http-nio-9080-exec-9] o.a.c.f.s.i.b.EndpointAddressValidator   : The endpointAddress value of http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize does not match any of the passive requestor values

 

 

I found a similar issue in the Talend Knowledgebase: https://community.talend.com/t5/Installation/Authentication-failed-in-Data-Stewardship/ta-p/36316

 

However, this is for Data Stewardship, but I still tried it out - no luck. Here is my curent config:

 

IAM: iam.properties:

 

 

Spoiler
tac.url=http://docker-dev-52.cgn.company.de:8080/org.talend.administrator-6.4.1/
tac.user-name=security@company.com
tac.password=XvHOgly6990XI3t4NVPN+g==
tac.application=DataPrep

# -----------------------------------------------------------------------------------
# IMPORTANT:
# Change of these 2 variables requires deleting both oidc and idp databases
# -----------------------------------------------------------------------------------
iam.host=docker-dev-52.cgn.company.de
iam.url=http://${iam.host}:9080

# IDP Settings
idp.url=${iam.url}/idp
idp.db.url=jdbc:h2:${CATALINA_HOME}/idp/idpdb;DB_CLOSE_DELAY=-1
idp.db.driverClassName=org.h2.Driver
idp.db.username=idp
idp.db.password=6zQsVb2TNlehmcBHS+E2qQ==
idp.db.defaultData=true
idp.db.platform=H2Dictionary

# OIDC Settings
oidc.url=${iam.url}/oidc
oidc.host=${iam.host}
oidc.issuer=accounts.talend.com
oidc.db.driverClassName=org.h2.Driver
oidc.db.url=jdbc:h2:${CATALINA_HOME}/oidc/oidcdb;DB_CLOSE_DELAY=-1
oidc.db.username=oidc
oidc.db.password=nL9phBqQQtb5iAu9f6uEUg==
oidc.db.databasePlatform=org.apache.openjpa.jdbc.sql.H2Dictionary
oidc.db.dialect=org.hibernate.dialect.H2Dialect
oidc.accessTokenLifetime=3600
oidc.dynRegService.initialAccessToken=S3hmqC5Q7SlGwmrfq190EA==
oidc.clientResources.pattern=file://${CATALINA_HOME}/clients/*.json

# STS Settings
sts.url=${iam.url}/sts-tac
sts.issuer=Fediz STS
sts.keystore.file=sts.jks
sts.keystore.password=l5j+/7FS8tiJUpU486CfxQ==
sts.keystore.alias=iam-sts-onpremise
sts.key.password=l5j+/7FS8tiJUpU486CfxQ==
sts.syncope.url=${iam.url}/sts
syncope.url=${iam.url}/syncope/rest
syncope.user-name=admin
syncope.password=S3hmqC5Q7SlGwmrfq190EA==

# SCIM Settings
security.oidc.client.keyUri=${oidc.url}/jwk/keys
security.oauth2.resource.tokenInfoUri=${oidc.url}/oauth2/introspect

# General Settings
log.path=${CATALINA_HOME}/logs
iam.config.encrypt=true

iam.additionalTLDs=lan,de,company,cgn,docker-dev-52

 

 

IAM / tdp-client.json:

 

Spoiler
{
  "post_logout_redirect_uris" : [ "http://docker-dev-52.cgn.company.de:9999", "http://localhost:9999", "http://127.0.0.1:9999" ],
  "grant_types" : [ "authorization_code", "refresh_token", "password" ],
  "scope" : "openid refreshToken",
  "client_secret" : "ef3+h0qm/1NSFfZX24TFAzIhWsvCjJmhlo6j05ktcho=",
  "redirect_uris" : [ "http://docker-dev-52.cgn.company.de:9999/signIn", "http://localhost:9999/signIn", "http://127.0.0.1:9999/signIn" ],
  "client_name" : "TDP DataPrep",
  "client_id" : "64xIVPxviKWSog"
}

 

 

 

DataPrep / application.properties:

 

 

Spoiler
#
# ============================================================================
# Copyright (C) 2006-2016 Talend Inc. - www.talend.com
#
# This source code is available under agreement available at
# https://github.com/Talend/data-prep/blob/master/LICENSE
#
# You should have received a copy of the agreement
# along with this program; if not, write to Talend SA
# 9 rue Pages 92150 Suresnes, France
#
# ============================================================================
#

#
# Security settings (using TAC):
#
# warning: the ending '/' is mandatory:
tac.url=http://docker-dev-52.cgn.company.de:8080/org.talend.administrator-6.4.1/

#
# Public IP:
# This is the public ip (or hostname) of the server hosting Data Prep server
#
public.ip=docker-dev-52.cgn.company.de
server.port=9999
iam.ip=docker-dev-52.cgn.company.de

# Async execution (leaves high value for large dataset support).
spring.mvc.async.request-timeout=300000

#
# Live datasets
#
# Data Prep will only list tasks with this prefix:
tac.task-prefix=dataprep_

# TAC user:
# It must have "Operation Manager" or "Designer" role, and have authorization on required projects to list tasks in "Talend job" datasets.
# It must have "Administrator" role in TAC, in order to list users and groups for sharing.
tac.user-name=dpadmin@company.de
tac.password=XvHOgly6990XI3t4NVPN+g==

#
# Mongodb settings
#
mongodb.host=localhost
mongodb.port=27017
mongodb.database=dataprep
mongodb.user=dataprep-user
mongodb.password=V4+wDTj9WTW5Qgr214HCXQ==
multi-tenancy.mongodb.active=true
# For more complex use cases, mongo.* configurations can be overridden by specifying URI directly:
mongodb.uri=mongodb://talend641-dev-mongodb:27017/dataprep

# Mongodb TLS settings
#
# mongodb.ssl=true
# mongodb.ssl.trust-store=/path/to/trust-store.jks
# mongodb.ssl.trust-store-password=trust-store-password
#
# HTTP TLS settings
#
# tls.key-store=/path/to/key-store.jks
# tls.key-store-password=key-store_password
# tls.trust-store=/path/to/trust-store.jks
# tls.trust-store-password=trust-store_password
# false to disable hostname verification
# tls.verify-hostname=false

security.provider=oauth2
security.token.secret=yZzHjE4PyAatbSY1/zj1vQ==
security.token.renew-after=30
security.token.invalid-after=3600

spring.profiles.active=server-standalone
spring.mvc.favicon.enabled=false

# Service documentation
service.documentation=false
service.documentation.name=Talend Data Preparation - API
service.documentation.description=This service exposes high level services that may involve services orchestration.
service.paths=api

# size limit for dataset in lines (if dataset.lines > limit, dataset is truncated)
dataset.records.limit=10000
dataset.local.file.size.limit=2000000000
dataset.imports=local,job,tcomp-JDBCDatastore,tcomp-SimpleFileIoDatastore,tcomp-SalesforceDatastore,tcomp-S3Datastore
dataset.list.limit=10

# Address of the data set service (set at runtime by unit tests depending on random port)
dataset.service.url=http://${public.ip}:${server.port}
transformation.service.url=http://${public.ip}:${server.port}
preparation.service.url=http://${public.ip}:${server.port}
fullrun.service.url=http://${public.ip}:${server.port}

# Configure all services for file storage
dataset.metadata.store=mongodb
# file or s3
preparation.store=mongodb
user.data.store=mongodb
folder.store=mongodb
upgrade.store=mongodb

# Cache management (location for cache and content storage)
content-service.store=local
content-service.store.local.path=data/

# Preparation service configuration (see preparation service)
preparation.store.remove.hours=24

# Lock on preparation (mongodb or none) delay in seconds
lock.preparation.store=mongodb
lock.preparation.delay=600

# Enable Hazelcast (true = enabled, false = disabled)
hazelcast.enabled=true

# Lucene index configuration
luceneIndexStrategy=singleton

#
# Asynchronous (full run / sampling) operations
#
# storage
execution.store=mongodb
# allowed concurrent runs
async.operation.concurrent.run=5

# TCOMP Server: deactivated if property is not present.
#tcomp.server.url=http://<place_tcomp_ip_here>:8989/tcomp

# hide some tcomp properties
# tcomp-JDBCDataset.sourceType.hide=true
# tcomp-JDBCDatastore.password.hide=true
tcomp-SimpleFileIoDatastore.kerberosPrincipal.default=${streams.kerberos.principal}
tcomp-SimpleFileIoDatastore.kerberosKeytab.default=${streams.kerberos.keytab_path}
tcomp-SimpleFileIoDataset.path.default=${streams.hdfs.server.url}

# remove test connection step from talend component form
tcomp-SimpleFileIoDatastore.test_connection.visible=false


# Full run task max execution time (max execution time for a full run in milliseconds)
# async.operation.watcher.ttl=3600000

# Max wait time when data prep waits for live data set input
# receivers.timeout=3600000
#
# Data Quality
#
# where indexes are extracted:
dataquality.indexes.file.location=data/data-quality/org.talend.dataquality.semantic
# display semantic types within dataprep UI
dataquality.semantic.list.enable=false
dataquality.server.url=<place_data-quality_server_url_here>

# to receive data quality updates
dataquality.semantic.update.enable=false
dataquality.event.store=mongodb
spring.cloud.stream.kafka.binder.brokers=<place_kafka_ip_here>
spring.cloud.stream.kafka.binder.zkNodes=<place_zookeeper_ip_here>
spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
spring.cloud.stream.kafka.binder.defaultZkPort=2181
spring.cloud.stream.bindings.input.destination=${MESSAGING_DOCUMENT_QUEUE:dictionary}
spring.cloud.stream.bindings.input.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQDocumentAction
spring.cloud.stream.bindings.input.group=${MESSAGING_CATEGORY_GROUP:dictionaryGroup}
spring.cloud.stream.bindings.category.destination=${MESSAGING_CATEGORY_QUEUE:category}
spring.cloud.stream.bindings.category.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQCategoryAction
spring.cloud.stream.bindings.category.group=${MESSAGING_REGEX_GROUP:dictionaryGroup}
spring.cloud.stream.bindings.regEx.destination=${MESSAGING_REGEX_QUEUE:regex}
spring.cloud.stream.bindings.regEx.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQCategoryAction
spring.cloud.stream.bindings.regEx.group=${MESSAGING_REGEX_GROUP:dictionaryGroup}
data.management.lucene.documents.folder=${dataquality.indexes.file.location}/index/dictionary
data.management.lucene.categories.folder=${dataquality.indexes.file.location}/category
data.management.receiving.folder=${dataquality.indexes.file.location}/index/received/
data.management.regex.folder=${dataquality.indexes.file.location}/regex

# Streams configuration
streams.enable=false
streams.flow.runner.url=http://<local machine ip>:<Big data preparation port>/v1
streams.kerberos.principal=<principal>
streams.kerberos.keytab_path=<keytab path>
streams.hdfs.server.url=hdfs://<host>:<port>/<filepath>

####################### SSO ##############
security.basic.enabled=false
security.oidc.client.expectedIssuer=accounts.talend.com
iam.license.url=http://${iam.ip}:9080/oidc/services
security.oidc.client.keyUri=http://${iam.ip}:9080/oidc/jwk/keys
security.oauth2.client.clientId=64xIVPxviKWSog
security.oauth2.client.clientSecret=1234567890qwertz
security.oidc.client.claimIssueAtTolerance=120
# security.oauth2.resource.serviceId=${PREFIX:}resource
security.oauth2.resource.tokenInfoUri=http://${iam.ip}:9080/oidc/oauth2/introspect
security.oauth2.resource.uri=/api/**,/folders/**,/datasets/**,/preparations/**,/transform/**,/version/**,/acl/**,/apply/**,/export,/export/**,/aggregate,/sampling/**,/receivers/**,/error,/docs,/datastores/**,/preparation/**
security.oauth2.resource.filter-order=3
security.oauth2.resource.tokenInfoUriCache.enabled=true
security.scim.cache.enabled=true
security.scim.enabled=true

security.oauth2.client.access-token-uri=http://${iam.ip}:9080/oidc/oauth2/token
security.oauth2.client.scope=openid refreshToken
security.oauth2.client.user-authorization-uri=http://${iam.ip}:9080/oidc/idp/authorize?prompt=none
security.oauth2.sso.login-use-forward=false
server.session.cookie.name=TDPSESSION
security.sessions=stateless
security.user.password=none

# SSO logout properties for dataprep API & Gateway
security.oidc.client.endSessionEndpoint=http://${iam.ip}:9080/oidc/idp/logout
security.oidc.client.logoutSuccessUrl=http://${public.ip}:${server.port}
security.oauth2.logout.uri=/signOut
security.oauth2.sso.login-path=/signIn

iam.scim.url=http://${iam.ip}:7777/scim/
####################### SSO ##############

gateway-api.service.url=http://${public.ip}:${server.port}
gateway-api.service.path=/gateway

zuul.servletPath=/gateway/upload

zuul.routes.dq.path=/gateway/dq/semanticservice/**
zuul.routes.dq.sensitiveHeaders=${zuul.sensitiveHeaders}
zuul.routes.dq.url=${dataquality.server.url}/
proxy.auth.routes.dq=oauth2

zuul.routes.api.path=/gateway/api/**
zuul.routes.api.sensitiveHeaders=${zuul.sensitiveHeaders}
zuul.routes.api.url=http://${public.ip}:${server.port}/api
proxy.auth.routes.api=oauth2

zuul.sensitiveHeaders=Cookie,Set-Cookie,Expires,X-Content-Type-Options,X-Xss-Protection,Cookie,X-Frame-Options,Cache-control,Pragma

zuul.host.socket-timeout-millis=300000
zuul.host.connect-timeout-millis=5000

############# LOGGING #############
## Path of the log file
logging.file=data/logs/app.log
## Level output pattern, uncomment to add the MDC user after level
logging.pattern.level=%5p [user %X{user}]
## Pattern used for file logging, uncomment to override Spring default
#logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p --- [%t] %-40.40logger{39} : %m%n%wEx
## Data-Prep loggers
logging.level.=WARN
logging.level.org.talend.dataprep=INFO
logging.level.org.talend.dataprep.api=INFO
logging.level.org.talend.dataprep.dataset=INFO
logging.level.org.talend.dataprep.preparation=INFO
logging.level.org.talend.dataprep.transformation=INFO
logging.level.org.talend.dataprep.fullrun=INFO
logging.level.org.talend.dataprep.api.dataquality=INFO
logging.level.org.talend.dataprep.configuration=INFO

To recap: 

Issue 1: Redirect to localhost instead of docker-dev-52.cgn.company.de

Issue 2: Login fails due to error (message) in IAM

 

Thanks for the help so far! 

Six Stars

Re: DataPrep 6.4.1 start fails in docker

Hi all,

 

I made some progress. Quick recap:

 

All components (DataPrep, TAC, IAM etc.) running in individual containers but on the same docker host. I have two issues now:

 

1. After opening the website of DataPrep, I get redirected to IAM to log in. However, it tries to redirect to a page on "localhost"like "http://localhost:9080/idp/federation?....". When I manually change the localhost to the hostname of my docker server, the redirect works and I can see the Talend IAM login page. When I enter the login credentials, I'll get the following error:

 

2018-02-23 19_50_31-Error.png

 

The IAM logs the following error (I just replaced the actual company name with 'company'):

 

 

2018-02-23 18:45:08.364 -ERROR [http-nio-9080-exec-6] o.a.c.f.s.i.b.EndpointAddressValidator   : The endpointAddress value of http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize does not match any of the passive requestor values

 

I found the following KnowledgeBase article: https://community.talend.com/t5/Installation/Authentication-failed-in-Data-Stewardship/ta-p/36316

 

However, this is for DataStewardship Consolse, but I still tried it out - doesn't work. Here are my current config files:

 

iam.properties:

Spoiler
tac.url=http://docker-dev-52.cgn.company.de:8080/org.talend.administrator-6.4.1/
tac.user-name=security@company.com
tac.password=XvHOgly6990XI3t4NVPN+g==
tac.application=DataPrep

# -----------------------------------------------------------------------------------
# IMPORTANT:
# Change of these 2 variables requires deleting both oidc and idp databases
# -----------------------------------------------------------------------------------
iam.host=docker-dev-52.cgn.company.de
iam.url=http://${iam.host}:9080

# IDP Settings
idp.url=${iam.url}/idp
idp.db.url=jdbc:h2:${CATALINA_HOME}/idp/idpdb;DB_CLOSE_DELAY=-1
idp.db.driverClassName=org.h2.Driver
idp.db.username=idp
idp.db.password=6zQsVb2TNlehmcBHS+E2qQ==
idp.db.defaultData=true
idp.db.platform=H2Dictionary

# OIDC Settings
oidc.url=${iam.url}/oidc
oidc.host=${iam.host}
oidc.issuer=accounts.talend.com
oidc.db.driverClassName=org.h2.Driver
oidc.db.url=jdbc:h2:${CATALINA_HOME}/oidc/oidcdb;DB_CLOSE_DELAY=-1
oidc.db.username=oidc
oidc.db.password=nL9phBqQQtb5iAu9f6uEUg==
oidc.db.databasePlatform=org.apache.openjpa.jdbc.sql.H2Dictionary
oidc.db.dialect=org.hibernate.dialect.H2Dialect
oidc.accessTokenLifetime=3600
oidc.dynRegService.initialAccessToken=S3hmqC5Q7SlGwmrfq190EA==
oidc.clientResources.pattern=file://${CATALINA_HOME}/clients/*.json

# STS Settings
sts.url=${iam.url}/sts-tac
sts.issuer=Fediz STS
sts.keystore.file=sts.jks
sts.keystore.password=l5j+/7FS8tiJUpU486CfxQ==
sts.keystore.alias=iam-sts-onpremise
sts.key.password=l5j+/7FS8tiJUpU486CfxQ==
sts.syncope.url=${iam.url}/sts
syncope.url=${iam.url}/syncope/rest
syncope.user-name=admin
syncope.password=S3hmqC5Q7SlGwmrfq190EA==

# SCIM Settings
security.oidc.client.keyUri=${oidc.url}/jwk/keys
security.oauth2.resource.tokenInfoUri=${oidc.url}/oauth2/introspect

# General Settings
log.path=${CATALINA_HOME}/logs
iam.config.encrypt=true

iam.additionalTLDs=lan,de,company,cgn,docker-dev-52

DataPrep application.properties:

 

Spoiler
#
# ============================================================================
# Copyright (C) 2006-2016 Talend Inc. - www.talend.com
#
# This source code is available under agreement available at
# https://github.com/Talend/data-prep/blob/master/LICENSE
#
# You should have received a copy of the agreement
# along with this program; if not, write to Talend SA
# 9 rue Pages 92150 Suresnes, France
#
# ============================================================================
#

#
# Security settings (using TAC):
#
# warning: the ending '/' is mandatory:
tac.url=http://docker-dev-52.cgn.company.de:8080/org.talend.administrator-6.4.1/

#
# Public IP:
# This is the public ip (or hostname) of the server hosting Data Prep server
#
public.ip=docker-dev-52.cgn.company.de
server.port=9999
iam.ip=docker-dev-52.cgn.company.de

# Async execution (leaves high value for large dataset support).
spring.mvc.async.request-timeout=300000

#
# Live datasets
#
# Data Prep will only list tasks with this prefix:
tac.task-prefix=dataprep_

# TAC user:
# It must have "Operation Manager" or "Designer" role, and have authorization on required projects to list tasks in "Talend job" datasets.
# It must have "Administrator" role in TAC, in order to list users and groups for sharing.
tac.user-name=dpadmin@company.de
tac.password=XvHOgly6990XI3t4NVPN+g==

#
# Mongodb settings
#
mongodb.host=localhost
mongodb.port=27017
mongodb.database=dataprep
mongodb.user=dataprep-user
mongodb.password=V4+wDTj9WTW5Qgr214HCXQ==
multi-tenancy.mongodb.active=true
# For more complex use cases, mongo.* configurations can be overridden by specifying URI directly:
mongodb.uri=mongodb://talend641-dev-mongodb:27017/dataprep

# Mongodb TLS settings
#
# mongodb.ssl=true
# mongodb.ssl.trust-store=/path/to/trust-store.jks
# mongodb.ssl.trust-store-password=trust-store-password
#
# HTTP TLS settings
#
# tls.key-store=/path/to/key-store.jks
# tls.key-store-password=key-store_password
# tls.trust-store=/path/to/trust-store.jks
# tls.trust-store-password=trust-store_password
# false to disable hostname verification
# tls.verify-hostname=false

security.provider=oauth2
security.token.secret=yZzHjE4PyAatbSY1/zj1vQ==
security.token.renew-after=30
security.token.invalid-after=3600

spring.profiles.active=server-standalone
spring.mvc.favicon.enabled=false

# Service documentation
service.documentation=false
service.documentation.name=Talend Data Preparation - API
service.documentation.description=This service exposes high level services that may involve services orchestration.
service.paths=api

# size limit for dataset in lines (if dataset.lines > limit, dataset is truncated)
dataset.records.limit=10000
dataset.local.file.size.limit=2000000000
dataset.imports=local,job,tcomp-JDBCDatastore,tcomp-SimpleFileIoDatastore,tcomp-SalesforceDatastore,tcomp-S3Datastore
dataset.list.limit=10

# Address of the data set service (set at runtime by unit tests depending on random port)
dataset.service.url=http://${public.ip}:${server.port}
transformation.service.url=http://${public.ip}:${server.port}
preparation.service.url=http://${public.ip}:${server.port}
fullrun.service.url=http://${public.ip}:${server.port}

# Configure all services for file storage
dataset.metadata.store=mongodb
# file or s3
preparation.store=mongodb
user.data.store=mongodb
folder.store=mongodb
upgrade.store=mongodb

# Cache management (location for cache and content storage)
content-service.store=local
content-service.store.local.path=data/

# Preparation service configuration (see preparation service)
preparation.store.remove.hours=24

# Lock on preparation (mongodb or none) delay in seconds
lock.preparation.store=mongodb
lock.preparation.delay=600

# Enable Hazelcast (true = enabled, false = disabled)
hazelcast.enabled=true

# Lucene index configuration
luceneIndexStrategy=singleton

#
# Asynchronous (full run / sampling) operations
#
# storage
execution.store=mongodb
# allowed concurrent runs
async.operation.concurrent.run=5

# TCOMP Server: deactivated if property is not present.
#tcomp.server.url=http://<place_tcomp_ip_here>:8989/tcomp

# hide some tcomp properties
# tcomp-JDBCDataset.sourceType.hide=true
# tcomp-JDBCDatastore.password.hide=true
tcomp-SimpleFileIoDatastore.kerberosPrincipal.default=${streams.kerberos.principal}
tcomp-SimpleFileIoDatastore.kerberosKeytab.default=${streams.kerberos.keytab_path}
tcomp-SimpleFileIoDataset.path.default=${streams.hdfs.server.url}

# remove test connection step from talend component form
tcomp-SimpleFileIoDatastore.test_connection.visible=false


# Full run task max execution time (max execution time for a full run in milliseconds)
# async.operation.watcher.ttl=3600000

# Max wait time when data prep waits for live data set input
# receivers.timeout=3600000
#
# Data Quality
#
# where indexes are extracted:
dataquality.indexes.file.location=data/data-quality/org.talend.dataquality.semantic
# display semantic types within dataprep UI
dataquality.semantic.list.enable=false
dataquality.server.url=<place_data-quality_server_url_here>

# to receive data quality updates
dataquality.semantic.update.enable=false
dataquality.event.store=mongodb
spring.cloud.stream.kafka.binder.brokers=<place_kafka_ip_here>
spring.cloud.stream.kafka.binder.zkNodes=<place_zookeeper_ip_here>
spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
spring.cloud.stream.kafka.binder.defaultZkPort=2181
spring.cloud.stream.bindings.input.destination=${MESSAGING_DOCUMENT_QUEUE:dictionary}
spring.cloud.stream.bindings.input.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQDocumentAction
spring.cloud.stream.bindings.input.group=${MESSAGING_CATEGORY_GROUP:dictionaryGroup}
spring.cloud.stream.bindings.category.destination=${MESSAGING_CATEGORY_QUEUE:category}
spring.cloud.stream.bindings.category.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQCategoryAction
spring.cloud.stream.bindings.category.group=${MESSAGING_REGEX_GROUP:dictionaryGroup}
spring.cloud.stream.bindings.regEx.destination=${MESSAGING_REGEX_QUEUE:regex}
spring.cloud.stream.bindings.regEx.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQCategoryAction
spring.cloud.stream.bindings.regEx.group=${MESSAGING_REGEX_GROUP:dictionaryGroup}
data.management.lucene.documents.folder=${dataquality.indexes.file.location}/index/dictionary
data.management.lucene.categories.folder=${dataquality.indexes.file.location}/category
data.management.receiving.folder=${dataquality.indexes.file.location}/index/received/
data.management.regex.folder=${dataquality.indexes.file.location}/regex

# Streams configuration
streams.enable=false
streams.flow.runner.url=http://<local machine ip>:<Big data preparation port>/v1
streams.kerberos.principal=<principal>
streams.kerberos.keytab_path=<keytab path>
streams.hdfs.server.url=hdfs://<host>:<port>/<filepath>

####################### SSO ##############
security.basic.enabled=false
security.oidc.client.expectedIssuer=accounts.talend.com
iam.license.url=http://${iam.ip}:9080/oidc/services
security.oidc.client.keyUri=http://${iam.ip}:9080/oidc/jwk/keys
security.oauth2.client.clientId=64xIVPxviKWSog
security.oauth2.client.clientSecret=1234567890qwertz
security.oidc.client.claimIssueAtTolerance=120
# security.oauth2.resource.serviceId=${PREFIX:}resource
security.oauth2.resource.tokenInfoUri=http://${iam.ip}:9080/oidc/oauth2/introspect
security.oauth2.resource.uri=/api/**,/folders/**,/datasets/**,/preparations/**,/transform/**,/version/**,/acl/**,/apply/**,/export,/export/**,/aggregate,/sampling/**,/receivers/**,/error,/docs,/datastores/**,/preparation/**
security.oauth2.resource.filter-order=3
security.oauth2.resource.tokenInfoUriCache.enabled=true
security.scim.cache.enabled=true
security.scim.enabled=true

security.oauth2.client.access-token-uri=http://${iam.ip}:9080/oidc/oauth2/token
security.oauth2.client.scope=openid refreshToken
security.oauth2.client.user-authorization-uri=http://${iam.ip}:9080/oidc/idp/authorize?prompt=none
security.oauth2.sso.login-use-forward=false
server.session.cookie.name=TDPSESSION
security.sessions=stateless
security.user.password=none

# SSO logout properties for dataprep API & Gateway
security.oidc.client.endSessionEndpoint=http://${iam.ip}:9080/oidc/idp/logout
security.oidc.client.logoutSuccessUrl=http://${public.ip}:${server.port}
security.oauth2.logout.uri=/signOut
security.oauth2.sso.login-path=/signIn

iam.scim.url=http://${iam.ip}:7777/scim/
####################### SSO ##############

gateway-api.service.url=http://${public.ip}:${server.port}
gateway-api.service.path=/gateway

zuul.servletPath=/gateway/upload

zuul.routes.dq.path=/gateway/dq/semanticservice/**
zuul.routes.dq.sensitiveHeaders=${zuul.sensitiveHeaders}
zuul.routes.dq.url=${dataquality.server.url}/
proxy.auth.routes.dq=oauth2

zuul.routes.api.path=/gateway/api/**
zuul.routes.api.sensitiveHeaders=${zuul.sensitiveHeaders}
zuul.routes.api.url=http://${public.ip}:${server.port}/api
proxy.auth.routes.api=oauth2

zuul.sensitiveHeaders=Cookie,Set-Cookie,Expires,X-Content-Type-Options,X-Xss-Protection,Cookie,X-Frame-Options,Cache-control,Pragma

zuul.host.socket-timeout-millis=300000
zuul.host.connect-timeout-millis=5000

############# LOGGING #############
## Path of the log file
logging.file=data/logs/app.log
## Level output pattern, uncomment to add the MDC user after level
logging.pattern.level=%5p [user %X{user}]
## Pattern used for file logging, uncomment to override Spring default
#logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p --- [%t] %-40.40logger{39} : %m%n%wEx
## Data-Prep loggers
logging.level.=WARN
logging.level.org.talend.dataprep=INFO
logging.level.org.talend.dataprep.api=INFO
logging.level.org.talend.dataprep.dataset=INFO
logging.level.org.talend.dataprep.preparation=INFO
logging.level.org.talend.dataprep.transformation=INFO
logging.level.org.talend.dataprep.fullrun=INFO
logging.level.org.talend.dataprep.api.dataquality=INFO
logging.level.org.talend.dataprep.configuration=INFO

Thanks for the help and let me know if you need more details!

Employee

Re: DataPrep 6.4.1 start fails in docker

Adding @MircoK's reply sent by private message:

 

Hm, for some reason my post wasn't saved. Well, here we go again :-) :



I managed to install and (almost) configure everything properly. Remember, all Talend components are in independent containers, though on the same docker server / host. At the moment, I have two issues:



1. When I go to the DataPrep Website, I get redirected to IAM in order to log in. However, I get redirected to "localhost", like that:

http://localhost:9080/idp/federation?wa=wsignin1.0&..........


When I change localhost to docker-dev-52.cgn.company.com, I get redirected to IAM. After logging in with the data prep user I have created in TAC, it leads me to issue 2:







The IAM log shows the following:



2018-02-23 19:41:20.878 -ERROR [http-nio-9080-exec-9] o.a.c.f.s.i.b.EndpointAddressValidator   : The endpointAddress value of http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize does not match any of the passive requestor values


I found a similar issue in the Talend Knowledgebase: https://community.talend.com/t5/Installation/Authentication-failed-in-Data-Stewardship/ta-p/36316



However, this is for Data Stewardship, but I still tried it out - no luck. Here is my curent config:



IAM: iam.properties:



tac.url=http://docker-dev-52.cgn.company.de:8080/org.talend.administrator-6.4.1/
tac.user-name=security@company.com
tac.password=XvHOgly6990XI3t4NVPN+g==
tac.application=DataPrep

# -----------------------------------------------------------------------------------
# IMPORTANT:
# Change of these 2 variables requires deleting both oidc and idp databases
# -----------------------------------------------------------------------------------
iam.host=docker-dev-52.cgn.company.de
iam.url=http://${iam.host}:9080

# IDP Settings
idp.url=${iam.url}/idp
idp.db.url=jdbc:h2:${CATALINA_HOME}/idp/idpdb;DB_CLOSE_DELAY=-1
idp.db.driverClassName=org.h2.Driver
idp.db.username=idp
idp.db.password=6zQsVb2TNlehmcBHS+E2qQ==
idp.db.defaultData=true
idp.db.platform=H2Dictionary

# OIDC Settings
oidc.url=${iam.url}/oidc
oidc.host=${iam.host}
oidc.issuer=accounts.talend.com
oidc.db.driverClassName=org.h2.Driver
oidc.db.url=jdbc:h2:${CATALINA_HOME}/oidc/oidcdb;DB_CLOSE_DELAY=-1
oidc.db.username=oidc
oidc.db.password=nL9phBqQQtb5iAu9f6uEUg==
oidc.db.databasePlatform=org.apache.openjpa.jdbc.sql.H2Dictionary
oidc.db.dialect=org.hibernate.dialect.H2Dialect
oidc.accessTokenLifetime=3600
oidc.dynRegService.initialAccessToken=S3hmqC5Q7SlGwmrfq190EA==
oidc.clientResources.pattern=file://${CATALINA_HOME}/clients/*.json

# STS Settings
sts.url=${iam.url}/sts-tac
sts.issuer=Fediz STS
sts.keystore.file=sts.jks
sts.keystore.password=l5j+/7FS8tiJUpU486CfxQ==
sts.keystore.alias=iam-sts-onpremise
sts.key.password=l5j+/7FS8tiJUpU486CfxQ==
sts.syncope.url=${iam.url}/sts
syncope.url=${iam.url}/syncope/rest
syncope.user-name=admin
syncope.password=S3hmqC5Q7SlGwmrfq190EA==

# SCIM Settings
security.oidc.client.keyUri=${oidc.url}/jwk/keys
security.oauth2.resource.tokenInfoUri=${oidc.url}/oauth2/introspect

# General Settings
log.path=${CATALINA_HOME}/logs
iam.config.encrypt=true

iam.additionalTLDs=lan,de,company,cgn,docker-dev-52

IAM / tdp-client.json:

{
  "post_logout_redirect_uris" : [ "http://docker-dev-52.cgn.company.de:9999", "http://localhost:9999", "http://127.0.0.1:9999" ],
  "grant_types" : [ "authorization_code", "refresh_token", "password" ],
  "scope" : "openid refreshToken",
  "client_secret" : "ef3+h0qm/1NSFfZX24TFAzIhWsvCjJmhlo6j05ktcho=",
  "redirect_uris" : [ "http://docker-dev-52.cgn.company.de:9999/signIn", "http://localhost:9999/signIn", "http://127.0.0.1:9999/signIn" ],
  "client_name" : "TDP DataPrep",
  "client_id" : "64xIVPxviKWSog"
}




DataPrep / application.properties:



#
# ============================================================================
# Copyright (C) 2006-2016 Talend Inc. - www.talend.com
#
# This source code is available under agreement available at
# https://github.com/Talend/data-prep/blob/master/LICENSE
#
# You should have received a copy of the agreement
# along with this program; if not, write to Talend SA
# 9 rue Pages 92150 Suresnes, France
#
# ============================================================================
#

#
# Security settings (using TAC):
#
# warning: the ending '/' is mandatory:
tac.url=http://docker-dev-52.cgn.company.de:8080/org.talend.administrator-6.4.1/

#
# Public IP:
# This is the public ip (or hostname) of the server hosting Data Prep server
#
public.ip=docker-dev-52.cgn.company.de
server.port=9999
iam.ip=docker-dev-52.cgn.company.de

# Async execution (leaves high value for large dataset support).
spring.mvc.async.request-timeout=300000

#
# Live datasets
#
# Data Prep will only list tasks with this prefix:
tac.task-prefix=dataprep_

# TAC user:
# It must have "Operation Manager" or "Designer" role, and have authorization on required projects to list tasks in "Talend job" datasets.
# It must have "Administrator" role in TAC, in order to list users and groups for sharing.
tac.user-name=dpadmin@company.de
tac.password=XvHOgly6990XI3t4NVPN+g==

#
# Mongodb settings
#
mongodb.host=localhost
mongodb.port=27017
mongodb.database=dataprep
mongodb.user=dataprep-user
mongodb.password=V4+wDTj9WTW5Qgr214HCXQ==
multi-tenancy.mongodb.active=true
# For more complex use cases, mongo.* configurations can be overridden by specifying URI directly:
mongodb.uri=mongodb://talend641-dev-mongodb:27017/dataprep

# Mongodb TLS settings
#
# mongodb.ssl=true
# mongodb.ssl.trust-store=/path/to/trust-store.jks
# mongodb.ssl.trust-store-password=trust-store-password
#
# HTTP TLS settings
#
# tls.key-store=/path/to/key-store.jks
# tls.key-store-password=key-store_password
# tls.trust-store=/path/to/trust-store.jks
# tls.trust-store-password=trust-store_password
# false to disable hostname verification
# tls.verify-hostname=false

security.provider=oauth2
security.token.secret=yZzHjE4PyAatbSY1/zj1vQ==
security.token.renew-after=30
security.token.invalid-after=3600

spring.profiles.active=server-standalone
spring.mvc.favicon.enabled=false

# Service documentation
service.documentation=false
service.documentation.name=Talend Data Preparation - API
service.documentation.description=This service exposes high level services that may involve services orchestration.
service.paths=api

# size limit for dataset in lines (if dataset.lines > limit, dataset is truncated)
dataset.records.limit=10000
dataset.local.file.size.limit=2000000000
dataset.imports=local,job,tcomp-JDBCDatastore,tcomp-SimpleFileIoDatastore,tcomp-SalesforceDatastore,tcomp-S3Datastore
dataset.list.limit=10

# Address of the data set service (set at runtime by unit tests depending on random port)
dataset.service.url=http://${public.ip}:${server.port}
transformation.service.url=http://${public.ip}:${server.port}
preparation.service.url=http://${public.ip}:${server.port}
fullrun.service.url=http://${public.ip}:${server.port}

# Configure all services for file storage
dataset.metadata.store=mongodb
# file or s3
preparation.store=mongodb
user.data.store=mongodb
folder.store=mongodb
upgrade.store=mongodb

# Cache management (location for cache and content storage)
content-service.store=local
content-service.store.local.path=data/

# Preparation service configuration (see preparation service)
preparation.store.remove.hours=24

# Lock on preparation (mongodb or none) delay in seconds
lock.preparation.store=mongodb
lock.preparation.delay=600

# Enable Hazelcast (true = enabled, false = disabled)
hazelcast.enabled=true

# Lucene index configuration
luceneIndexStrategy=singleton

#
# Asynchronous (full run / sampling) operations
#
# storage
execution.store=mongodb
# allowed concurrent runs
async.operation.concurrent.run=5

# TCOMP Server: deactivated if property is not present.
#tcomp.server.url=http://<place_tcomp_ip_here>:8989/tcomp

# hide some tcomp properties
# tcomp-JDBCDataset.sourceType.hide=true
# tcomp-JDBCDatastore.password.hide=true
tcomp-SimpleFileIoDatastore.kerberosPrincipal.default=${streams.kerberos.principal}
tcomp-SimpleFileIoDatastore.kerberosKeytab.default=${streams.kerberos.keytab_path}
tcomp-SimpleFileIoDataset.path.default=${streams.hdfs.server.url}

# remove test connection step from talend component form
tcomp-SimpleFileIoDatastore.test_connection.visible=false


# Full run task max execution time (max execution time for a full run in milliseconds)
# async.operation.watcher.ttl=3600000

# Max wait time when data prep waits for live data set input
# receivers.timeout=3600000
#
# Data Quality
#
# where indexes are extracted:
dataquality.indexes.file.location=data/data-quality/org.talend.dataquality.semantic
# display semantic types within dataprep UI
dataquality.semantic.list.enable=false
dataquality.server.url=<place_data-quality_server_url_here>

# to receive data quality updates
dataquality.semantic.update.enable=false
dataquality.event.store=mongodb
spring.cloud.stream.kafka.binder.brokers=<place_kafka_ip_here>
spring.cloud.stream.kafka.binder.zkNodes=<place_zookeeper_ip_here>
spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
spring.cloud.stream.kafka.binder.defaultZkPort=2181
spring.cloud.stream.bindings.input.destination=${MESSAGING_DOCUMENT_QUEUE:dictionary}
spring.cloud.stream.bindings.input.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQDocumentAction
spring.cloud.stream.bindings.input.group=${MESSAGING_CATEGORY_GROUP:dictionaryGroup}
spring.cloud.stream.bindings.category.destination=${MESSAGING_CATEGORY_QUEUE:category}
spring.cloud.stream.bindings.category.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQCategoryAction
spring.cloud.stream.bindings.category.group=${MESSAGING_REGEX_GROUP:dictionaryGroup}
spring.cloud.stream.bindings.regEx.destination=${MESSAGING_REGEX_QUEUE:regex}
spring.cloud.stream.bindings.regEx.content-type=application/x-java-object;type=org.talend.dataquality.semantic.model.DQCategoryAction
spring.cloud.stream.bindings.regEx.group=${MESSAGING_REGEX_GROUP:dictionaryGroup}
data.management.lucene.documents.folder=${dataquality.indexes.file.location}/index/dictionary
data.management.lucene.categories.folder=${dataquality.indexes.file.location}/category
data.management.receiving.folder=${dataquality.indexes.file.location}/index/received/
data.management.regex.folder=${dataquality.indexes.file.location}/regex

# Streams configuration
streams.enable=false
streams.flow.runner.url=http://<local machine ip>:<Big data preparation port>/v1
streams.kerberos.principal=<principal>
streams.kerberos.keytab_path=<keytab path>
streams.hdfs.server.url=hdfs://<host>:<port>/<filepath>

####################### SSO ##############
security.basic.enabled=false
security.oidc.client.expectedIssuer=accounts.talend.com
iam.license.url=http://${iam.ip}:9080/oidc/services
security.oidc.client.keyUri=http://${iam.ip}:9080/oidc/jwk/keys
security.oauth2.client.clientId=64xIVPxviKWSog
security.oauth2.client.clientSecret=1234567890qwertz
security.oidc.client.claimIssueAtTolerance=120
# security.oauth2.resource.serviceId=${PREFIX:}resource
security.oauth2.resource.tokenInfoUri=http://${iam.ip}:9080/oidc/oauth2/introspect
security.oauth2.resource.uri=/api/**,/folders/**,/datasets/**,/preparations/**,/transform/**,/version/**,/acl/**,/apply/**,/export,/export/**,/aggregate,/sampling/**,/receivers/**,/error,/docs,/datastores/**,/preparation/**
security.oauth2.resource.filter-order=3
security.oauth2.resource.tokenInfoUriCache.enabled=true
security.scim.cache.enabled=true
security.scim.enabled=true

security.oauth2.client.access-token-uri=http://${iam.ip}:9080/oidc/oauth2/token
security.oauth2.client.scope=openid refreshToken
security.oauth2.client.user-authorization-uri=http://${iam.ip}:9080/oidc/idp/authorize?prompt=none
security.oauth2.sso.login-use-forward=false
server.session.cookie.name=TDPSESSION
security.sessions=stateless
security.user.password=none

# SSO logout properties for dataprep API & Gateway
security.oidc.client.endSessionEndpoint=http://${iam.ip}:9080/oidc/idp/logout
security.oidc.client.logoutSuccessUrl=http://${public.ip}:${server.port}
security.oauth2.logout.uri=/signOut
security.oauth2.sso.login-path=/signIn

iam.scim.url=http://${iam.ip}:7777/scim/
####################### SSO ##############

gateway-api.service.url=http://${public.ip}:${server.port}
gateway-api.service.path=/gateway

zuul.servletPath=/gateway/upload

zuul.routes.dq.path=/gateway/dq/semanticservice/**
zuul.routes.dq.sensitiveHeaders=${zuul.sensitiveHeaders}
zuul.routes.dq.url=${dataquality.server.url}/
proxy.auth.routes.dq=oauth2

zuul.routes.api.path=/gateway/api/**
zuul.routes.api.sensitiveHeaders=${zuul.sensitiveHeaders}
zuul.routes.api.url=http://${public.ip}:${server.port}/api
proxy.auth.routes.api=oauth2

zuul.sensitiveHeaders=Cookie,Set-Cookie,Expires,X-Content-Type-Options,X-Xss-Protection,Cookie,X-Frame-Options,Cache-control,Pragma

zuul.host.socket-timeout-millis=300000
zuul.host.connect-timeout-millis=5000

############# LOGGING #############
## Path of the log file
logging.file=data/logs/app.log
## Level output pattern, uncomment to add the MDC user after level
logging.pattern.level=%5p [user %X{user}]
## Pattern used for file logging, uncomment to override Spring default
#logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p --- [%t] %-40.40logger{39} : %m%n%wEx
## Data-Prep loggers
logging.level.=WARN
logging.level.org.talend.dataprep=INFO
logging.level.org.talend.dataprep.api=INFO
logging.level.org.talend.dataprep.dataset=INFO
logging.level.org.talend.dataprep.preparation=INFO
logging.level.org.talend.dataprep.transformation=INFO
logging.level.org.talend.dataprep.fullrun=INFO
logging.level.org.talend.dataprep.api.dataquality=INFO
logging.level.org.talend.dataprep.configuration=INFO
To recap: 

Issue 1: Redirect to localhost instead of docker-dev-52.cgn.company.de

Issue 2: Login fails due to error (message) in IAM



Thanks for the help so far!

 

 

@asharma@fhuaulme, @smallet, can you help on that one?

 

BTW, @MircoK: thank you for pointing out the gap in the product documentation regarding the IAM <=> TAC configuration. I've brought it up to our documentation team and it will get fixed soon enough.

 

Regards,

 

Gwendal

Six Stars

Re: DataPrep 6.4.1 start fails in docker

@gvaznunes Funny, now all posts appeared Smiley Very Happy. Feel free to delete the duplicate ones Smiley Happy. Btw. I will reach out to you next week with some feedback to the documentation. Now that I know that the feedback actually is getting heard, I'm more than glad to help improving the docs! 

 

I really appreciate the direct communication with the team here, this is a big pro of Talend!

Employee

Re: DataPrep 6.4.1 start fails in docker

Wow, that is awkward. Let's keep all the duplicated posts, it could help the Community team understand what happened.

 

And yes, we truly do listen - there is nothing more valuable than direct user feedback! So thanks again for your help Smiley Happy

Six Stars

Re: DataPrep 6.4.1 start fails in docker

I just finished the Data-Stewardship docker container and experience the exact same issue as mentioned above:

 

1. The redirect for login goes to "localhost" instead of the IAM URL (docker-dev-52.cgn.company.de)

2. When I fix the redirect manually I get to the login page, try to login and get the error in IAM: 

 The endpointAddress value of http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize does not match any of the passive requestor values

Now I'm really stuck, can't proceed with our installation without DataPrep and DataStewardship Smiley Sad. Appreciate any help!

Employee

Re: DataPrep 6.4.1 start fails in docker

 

This should help with the second error you are seeing - Did you try deleting the "idp" and "oidc" databases after changing the "oidc.host" value to "docker-dev-52.cgn.company.de"? The "passive requestor value" warning is when "oidc.host" does not match the hostname, however it appears that you have "oidc.host" configured correctly. After changing "oidc.host" it's necessary to delete the databases though as per the comment in the configuration file. Could you try doing this and see if step "2" works?

 

Colm.

Six Stars

Re: DataPrep 6.4.1 start fails in docker

Thanks, that actually helped. Of course, I read that comment but I expected that due to the fact that I'm using Docker, with each try I have a clean database anyway. This is basically true. But I forgot that during the build-process of the docker image, I start IAM once. So I already have a "bad" database in my initial image. I have fixed that and the mentioned error is gone!

 

However, this led me two the next error:

 

docker_dataprep_error_2.png

 

IAM Log:

2018-02-27 12:25:43.068 - INFO [http-nio-9080-exec-7] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 2
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://docker-dev-52.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=IharjG
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[JSESSIONID=6A038137ECA54D9D6345DB3638967D03; TDPSESSION=node0hdk5nkc4l9f1lfblfvceiuzq0.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:25:43.069 - WARN [http-nio-9080-exec-7] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended
27-Feb-2018 12:26:32.164 INFO [http-nio-9080-exec-1] org.apache.cxf.fediz.tomcat8.FederationAuthenticator.authenticate No valid principal found in existing session. Redirecting to IDP
2018-02-27 12:26:51.708 - INFO [http-nio-9080-exec-7] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 3
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://docker-dev-52.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=7q69pP
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[TDPSESSION=node01ledn86cvzzrs11x2l6bgvc6b11.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:26:51.709 - WARN [http-nio-9080-exec-7] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended

 

I have also created a Data Stewardship container. The error I get there is a bit different:

 

data_steward_error.png

 

IAM Log:

2018-02-27 12:34:11.230 - INFO [http-nio-9080-exec-5] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 4
Address: http://docker-dev-52.cgn.company.de:9080/oidc/idp/authorize?consent=none&client_id=tl6K6ac7tSE-LQ&redirect_uri=http://docker-dev-53.cgn.company.de:8080/login&response_type=code&scope=openid%20refreshToken&state=9pAItA
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], cookie=[JSESSIONID=6A038137ECA54D9D6345DB3638967D03; TDPSESSION=node0hdk5nkc4l9f1lfblfvceiuzq0.node0], host=[docker-dev-52.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36]}
--------------------------------------
2018-02-27 12:34:11.231 - WARN [http-nio-9080-exec-5] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended
2018-02-27 12:34:11.237 - WARN [http-nio-9080-exec-5] o.t.i.o.a.BadRequestExceptionMapper      : An OAuth2 BadRequestException occured: HTTP 400 Bad Request
javax.ws.rs.BadRequestException: HTTP 400 Bad Request
        at org.apache.cxf.jaxrs.utils.SpecExceptions.toBadRequestException(SpecExceptions.java:84)
        at org.apache.cxf.jaxrs.utils.ExceptionUtils.toBadRequestException(ExceptionUtils.java:121)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:134)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:122)
        at org.apache.cxf.rs.security.oauth2.services.AbstractOAuthService.reportInvalidRequestError(AbstractOAuthService.java:116)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.validateRedirectUri(RedirectionBasedGrantService.java:459)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.startAuthorization(RedirectionBasedGrantService.java:136)
        at org.apache.cxf.rs.security.oauth2.services.RedirectionBasedGrantService.authorize(RedirectionBasedGrantService.java:96)
        at org.apache.cxf.rs.security.oauth2.services.AuthorizationService.authorize(AuthorizationService.java:58)
        [....] cut off 

 

Any ideas Smiley Happy? Thanks for the help so far!

Employee

Re: DataPrep 6.4.1 start fails in docker

In terms of the TDS error, the redirect_uri from the log snippet is:

redirect_uri=http://docker-dev-53.cgn.company.de:8080/login

Does this match what is configured in clients/tds-client.json? e.g.:

"redirect_uris" : [ "http://my-machine:19999/login", "http://localhost:19999/login", "http://127.0.0.1:19999/login" ],

 See the following documentation for more information:

 

https://help.talend.com/reader/vuI_X~V6unFjTgNxRMPcLw/5q~UoTnzcjlhNT0CB1~RJQ

Six Stars

Re: DataPrep 6.4.1 start fails in docker

I'm working so long on that setup that I start being disregardful ;-). This was actually the problem. After that, I got some other issues, but these were caused by a missing portmapping (8090) or wrong TAC credentials. Now, it seems to almost work. I can add campaigns or data models. Only when I get to "Semantic Types", I get the following issue:

 

Spoiler
GATEWAY 2018-02-27 16:46:13,520  WARN [io-8080-exec-11] o.s.c.n.z.filters.post.SendErrorFilter   : Error during filtering
com.netflix.zuul.exception.ZuulException: Connection reset
        at org.springframework.cloud.netflix.zuul.util.ZuulRuntimeException.<init>(ZuulRuntimeException.java:33)
        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:200)
        at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
        at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193)
        at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157)
        at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118)
        at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)
        at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)
        at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)
        at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)
        at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:108)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.talend.iam.swagger.autoconfigure.FixSwagger2Filter.doFilter(FixSwagger2Filter.java:83)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.security.oauth2.client.filter.OAuth2ClientContextFilter.doFilter(OAuth2ClientContextFilter.java:60)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:167)
        at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:145)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
        at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
        at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forwardRequest(SimpleHostRoutingFilter.java:385)
        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forward(SimpleHostRoutingFilter.java:304)
        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:195)
        ... 121 common frames omitted

I will further anlayze that issue by myself as I think I could have missed something in the documentation. I just wanted to give you an update already. Thanks!

 

 

Edit: For DataPrep, the issue is still there but I will check if any of the TDS issues might also be the case for DataPrep.

Six Stars

Re: DataPrep 6.4.1 start fails in docker

Hi all,

 

good news: I got (almost) everything running! Most of the issues were caused by invalid or missing config. The redirecting issue for example can be fixed by changing the IP address in the fediz config file.

 

bad news: After 6.4.1 was running, we received our license for 6.5.1. The update itself was pretty easy and took only ~30 minutes (thanks to Docker) but I do have a problem with Talend Data Prep now. After logging in, I receive the following message: 2018-03-12 21_52_56-Clipboard.png

 

The log message from TDP:

 

Spoiler
2018-03-12 20:38:19.508 ERROR [user ] 123 --- [qtp1304345230-17] a.o.c.AuthenticationFailureEventListener : Authentication error for principal: org.springframework.security.oauth2.client.filter.FailedOAuthClientAuthentication@19d13835: Principal: UNKNOWN; Credentials: [PROTECTED]; Authenticated: false; Details: null; Not granted any authorities: {}

org.springframework.security.authentication.BadCredentialsException: Could not obtain user details from token
        at org.talend.iam.security.oidc.client.filter.OidcClientAuthenticationProcessingFilter.attemptAuthentication(OidcClientAuthenticationProcessingFilter.java:110) ~[oidc-client-1.0.0-20171205153406-g1852b42.jar:na]
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212) [spring-security-web-4.2.2.RELEASE.jar:4.2.2.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.2.RELEASE.jar:4.2.2.RELEASE]
       
[ausgeschnitten, da sonst 20.000 chars überschritten werden]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: org.springframework.security.oauth2.common.exceptions.InvalidTokenException: Invalid id_token (iss claim) unexpected issuer : http://docker-dev-54.cgn.company.de:9080/oidc at org.talend.iam.security.oidc.client.token.validator.DefaultOidcIdTokenValidator.validateOpenId(DefaultOidcIdTokenValidator.java:102) ~[oidc-client-1.0.0-20171205153406-g1852b42.jar:na] at org.talend.iam.security.oidc.client.token.validator.DefaultOidcIdTokenValidator.validate(DefaultOidcIdTokenValidator.java:82) ~[oidc-client-1.0.0-20171205153406-g1852b42.jar:na] at org.talend.iam.security.oidc.client.token.DefaultTokenAuthenticationServices.validate(DefaultTokenAuthenticationServices.java:54) ~[oidc-client-1.0.0-20171205153406-g1852b42.jar:na] at org.talend.iam.security.oidc.client.token.DefaultTokenAuthenticationServices.loadAuthentication(DefaultTokenAuthenticationServices.java:39) ~[oidc-client-1.0.0-20171205153406-g1852b42.jar:na] at org.talend.iam.security.oidc.client.filter.OidcClientAuthenticationProcessingFilter.attemptAuthentication(OidcClientAuthenticationProcessingFilter.java:101) ~[oidc-client-1.0.0-20171205153406-g1852b42.jar:na] ... 61 common frames omitted

Log message from IAM (oidc.log):

 

Spoiler
2018-03-12 20:51:52.820 - INFO [http-nio-9080-exec-8] o.a.c.interceptor.LoggingInInterceptor   : Inbound Message
----------------------------
ID: 5
Address: http://docker-dev-54.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://talend651-dev-tdp.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=oN4LBq
Encoding: UTF-8
Http-Method: GET
Content-Type:
Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], host=[docker-dev-54.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36]}
--------------------------------------
2018-03-12 20:51:52.821 - WARN [http-nio-9080-exec-8] o.a.c.r.s.o.s.AbstractOAuthService       : Unsecure HTTP, HTTPS is recommended
2018-03-12 20:51:52.924 - WARN [http-nio-9080-exec-5] o.a.c.j.i.WebApplicationExceptionMapper  : javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
        at org.apache.cxf.jaxrs.utils.SpecExceptions.toNotAuthorizedException(SpecExceptions.java:94)
        at org.apache.cxf.jaxrs.utils.ExceptionUtils.toNotAuthorizedException(ExceptionUtils.java:137)
        at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.reportInvalidClient(AbstractTokenService.java:278)
        at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.reportInvalidClient(AbstractTokenService.java:273)
        at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.getAndValidateClientFromIdAndSecret(AbstractTokenService.java:133)
        at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.getClientFromBasicAuthScheme(AbstractTokenService.java:155)
        at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.authenticateClientIfNeeded(AbstractTokenService.java:96)
        at org.apache.cxf.rs.security.oauth2.services.AccessTokenService.handleTokenRequest(AccessTokenService.java:98)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       
[ausgeschnitten]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) 2018-03-12 20:53:44.924 - INFO [http-nio-9080-exec-6] f.s.w.FederationAuthenticationEntryPoint : Redirecting to IDP: http://docker-dev-54.cgn.company.de:9080/idp/federation?wa=wsignin1.0&wreply=http%3A%2F%2Fdocker-dev-54.cgn.company.de%3A9080%2Foidc%2Flogin&wtrealm=urn%3Aorg%3Aapache%3Acxf%3Afediz%3Aoidc&wct=2018-03-12T20%3A53%3A44.924Z&wctx=a4e6802c-8f39-49c4-bb4b-72efb0c03b28&auth_app=%3Capplication+xmlns%3D%22http%3A%2F%2Fiam.talend.org%2Fapplication%22%3Etdp%3C%2Fapplication%3E 2018-03-12 20:53:49.397 - INFO [http-nio-9080-exec-7] o.a.c.interceptor.LoggingInInterceptor : Inbound Message ---------------------------- ID: 6 Address: http://docker-dev-54.cgn.company.de:9080/oidc/idp/authorize?prompt=none&client_id=64xIVPxviKWSog&redirect_uri=http://talend651-dev-tdp.cgn.company.de:9999/signIn&response_type=code&scope=openid%20refreshToken&state=gH2pav Encoding: UTF-8 Http-Method: GET Content-Type: Headers: {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8], accept-encoding=[gzip, deflate], accept-language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7], connection=[keep-alive], Content-Type=[null], host=[docker-dev-54.cgn.company.de:9080], upgrade-insecure-requests=[1], user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36]} -------------------------------------- 2018-03-12 20:53:49.398 - WARN [http-nio-9080-exec-7] o.a.c.r.s.o.s.AbstractOAuthService : Unsecure HTTP, HTTPS is recommended 2018-03-12 20:53:49.479 - WARN [http-nio-9080-exec-9] o.a.c.j.i.WebApplicationExceptionMapper : javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized at org.apache.cxf.jaxrs.utils.SpecExceptions.toNotAuthorizedException(SpecExceptions.java:94) at org.apache.cxf.jaxrs.utils.ExceptionUtils.toNotAuthorizedException(ExceptionUtils.java:137) at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.reportInvalidClient(AbstractTokenService.java:278) at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.reportInvalidClient(AbstractTokenService.java:273) at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.getAndValidateClientFromIdAndSecret(AbstractTokenService.java:133) at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.getClientFromBasicAuthScheme(AbstractTokenService.java:155) at org.apache.cxf.rs.security.oauth2.services.AbstractTokenService.authenticateClientIfNeeded(AbstractTokenService.java:96) at org.apache.cxf.rs.security.oauth2.services.AccessTokenService.handleTokenRequest(AccessTokenService.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:191) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
[ausgeschnitten]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

The other logfiles didn't show any errors as far as I can see.

 

I checked all settings but can't figure out why it doesn't work in 6.5.1. Talend Data Steward funktioniert auf 6.5.1....Jemand eine Idee?

Employee

Re: DataPrep 6.4.1 start fails in docker

What is your issuer attribute configuration in this file

  • Open the <installation_path>/iam/apache-tomcat/conf/fediz_config.xml file

    <issuer>http://<iam_urlSmiley Tongueort>/idp/federation</issuer> ?
Six Stars

Re: DataPrep 6.4.1 start fails in docker

Hi,

 

the value is:

 

<issuer>http://docker-dev-54.cgn.company.de:9080/idp/federation</issuer>
Employee

Re: DataPrep 6.4.1 start fails in docker

That looks good. So after 6.5.1 upgrade, TDS is working fine for you, you only have issues with TDP, correct?

Was any password changed for the user you are using to log in to TDP or for TAC security administrator user? Since I see 401 Unauthorized errors or any Role changed for the user you are using to use TDP ?

Six Stars

Re: DataPrep 6.4.1 start fails in docker

Yes, correct. After the 6.5.1 upgrade TDP doesn't work, TDS works. Since all Talend components get automatically configured during the creation of the Docker images, nothing should have changed. Only the Talend installation files are different of course (6.5.1 instead of 6.4.1). The TAC database (MySQL) has been migrated as advised (opened with TAC 6.5.1 and used the "migration task"). We didn't change the users.

 

We use a user called "dpadmin@company.com" which has all privileges (Secruity Admin, Admin etc.). We use this user in a call config files.

 

Since the TDS connection works, the user itself can't be the reason, right? You are right that the error is 401 Unauthorized, but later in the TDP logs I see also the following error:

 

Caused by: org.springframework.security.oauth2.common.exceptions.InvalidTokenException: Invalid id_token (iss claim) unexpected issuer : http://docker-dev-54.cgn.company.de:9080/oidc

 

 

Employee

Re: DataPrep 6.4.1 start fails in docker

What is the value in TDP 'config/application.properties' for:

* iam.ip

* security.oidc.client.expectedIssuer

The latter needs to match the issuer in the token created by Talend IAM, for your case:

http://docker-dev-54.cgn.company.de:9080/oidc

 

Six Stars

Re: DataPrep 6.4.1 start fails in docker

Hi,

 

Settings are:

 

 

iam.ip=docker-dev-54.cgn.company.de
security.oidc.client.expectedIssuer=http://${iam.ip}:8080/oidc

 

After a while, I saw the issue...The port of security.oidc.client.expectedIssuer is 8080 by default, I'm pretty sure it was 9080 by default in 6.4.1 as we didn't change the port. After changing the port to 9080, the login works! Thanks for the help!

 

But this brings me to another TDP issue which we also had in 6.4.1, but I didn't have the time to post it yet. It is regarding the component catalog service, located in {TDP}/services. When we try to start that service, it starts listening on port 9999 which is of course blocked by DataPrep itself. This is the config of the tcomp service:

 

#port exposed for the tcomp endpoints
server.port=8989
#context path for the tcomp endpoints
server.contextPath=${SERVER_CONTEXT_PATH:/tcomp}
# see http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-customize-the-jackson-objectmapper
spring.jackson.serialization-inclusion=non_empty
#directory containing the Hadoop configuration XML files.  If the environment
#variable HADOOP_CONF_DIR is set, it has priority over this value.
#hadoop.conf.dir=/opt/shared/hadoop/conf/

#location of the krb5 configuration file.  If the environment variable
#KRB5_CONFIG is set, it has priority over this value.
#krb5.config=/etc/krb5.conf

#Location of the remote maven repository for getting components and dependencies
#if the environment variable PAX_MVN_REPO is set it has priority over this value.
#the format of this property is exactly the same as the org.ops4j.pax.url.mvn.repositories defined in
#     https://ops4j1.jira.com/wiki/display/paxurl/Mvn+Protocol
#pax.mvn.repo=https://artifacts-oss.talend.com/nexus/content/repositories/TalendOpenSourceSnapshot/@snapshots@id=talend,http://central.maven.org/maven2@id=maven

#mvn uri to find the list of components to load, this will be used if not file named components.list is found on the classpath
component.default.config.mvn.url=${MVN_CONFIG_URL:mvn:org.talend.components/components-maven-repo/0.21.0/zip/config}

The port is set to 8989. Appreciate the help!

 

Best regards,

MK