Question

How do you generate a trace of the HTTP requests executed by the Talend Studio without using a third-party tool?

Answer

1. Create a logging.properties file (in the c:\temp folder for example) containing the following lines:

.level=FINEST
handlers = java.util.logging.FileHandler
java.util.logging.FileHandler.pattern = c:/temp/debug.txt
java.util.logging.FileHandler.limit = 20480KB
java.util.logging.FileHandler.count = 10
java.util.logging.FileHandler.append = true
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
 

2. Create a log4j.properties file (in the c:\temp folder for example) containing the following lines:

log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
log4j.logger.org.apache.http=DEBUG
log4j.logger.org.apache.http.wire=DEBUG

Note: the trace generated does not contain all the body of the HTTP requests. If you want to log the body of the HTTP requests and responses, use a HTTP proxy, such as Fiddler.

3. Start Talend Studio  by using the following commands:

On Windows : from a CMD window , execute the following 2 commands :

set _JAVA_OPTIONS=-Dlog4j.debug -Dlog4j.configuration=file:"c:\temp\log4j.properties" -Djava.util.logging.config.file=c:\temp\logging.properties
Talend-Studio-win-x86_64.exe --talendDebug > studio_debug.txt 2>&1

On Linux : from a shell terminal , execute the following 2 commands :

export _JAVA_OPTIONS='-Dlog4j.debug -Dlog4j.configuration=file:"/tmp/log4j.properties" -Djava.util.logging.config.file=/tmp/logging.properties'
Talend-Studio-linux-gtk-x86_64 --talendDebug > studio_debug.txt 2>&1

NOTE : On Linux , "adapt" the file  logging.properties :

java.util.logging.FileHandler.pattern = /tmp/debug.txt

Problem Description

After migrating from Talend 6.1.1 to Talend 6.3.1, a decrease in performance is observed when Jobs are writing data into a MS SQL database server.

This issue occurs when using the Talend 6.3.1 tMSSqlOutput component: with this component, writing data to the MS SQL server can take six times longer than with Talend 6.1.

Root Cause

The cause of decreased performance is due to the bug TDI-38197.

Solution

From Talend Studio 6.3.1 apply the patch: Patch_20170516_TPS-1951_v1-6.3.1, available from Talend Support.

To apply the patch manually, perform the steps indicated in the readme.txt included in the patch file (Patch_20170516_TPS-1951_v1-6.3.1.zip).

Problem Description

Opening an existing SVN project in Talend Studio, results in the following error message:

The Studio error log displays the following message:

!ENTRY org.talend.platform.logging 2 0 2018-08-23 10:27:04.344
!MESSAGE 2018-08-23 10:27:04,344 WARN org.talend.commons.exception.CommonExceptionHandler - org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1DiagnosticWrappedException: org.eclipse.emf.ecore.xmi.FeatureNotFoundException: Feature 'bigData' not found.

Root Cause

This issue may happen when opening an existing project with a higher version of Talend, for example, opening a Talend 5.6 project in Talend 6.4. Opening the project in a higher version generates new binaries that might conflict with the existing ones.

Solutions

To resolve this issue, go back to the SVN project that wasn't changed for the higher version of Studio, by performing the following steps:

1. Identify the version you want to go back to.
2. Dump that revision into a new dump file.
3. Load the new dump file into a new repository.
4. Replace the old repository with the new repository.

Problem description

Set up Talend Administration Center and SSL enabled on AWS RDS MySQL to transmit their data over SSL on a network from the DB Config page of Talend Administration Center.

Root cause

Talend Administration Center and AWS RDS MySQL are not securely transmitting their data over the network by default.

Solution

Add the JDBC URL with additional parameters to access the MySQL server that was SSL encrypted.

jdbc:mysql://<Endpoint>:3306/<DB_Name>?verifyServerCertificate=true&useSSL=true&requireSSL=true&trustCertificateKeyStoreUrl=file:///<Path_to_keystore>&trustCertificateKeyStorePassword=password

The command to generate keystore using the AWS bundled certificate using Java keytool is as follows:

keytool -import -alias <alias_name> -file rds-combined-ca-bundle.pem -keystore <name_of_keystore>

Example:

keytool -import -alias rdsmysql -file rds-combined-ca-bundle.pem -keystore rdskeystore

Note: This works on 7.2.1 only when you apply the TPS-3375 patch. Contact Talend Support to request the patch.

Tips: If you're still experiencing issues, add Djavax.net.debug=SSL in the setenv.sh file, to find out which certs are used by the configuration and to avoid the conflicts.

Problem Description

You may have multiple instances of Talend Administration Center (TAC) representing a development layer, for example dev, test, and production. Each TAC has its own repository database (or more precisely, a schema in a shared database), so you need to enter each configuration manually in each TAC.

Is it possible for some repository tables to use views for the TACs in layers test and production? For example:

• Projects
• Project References
• Dashboard connections
• Users
• UserRoleReference

These views get their data from the original table of the TAC in layer dev. Each change you make there will be "published" (or visible) to the other two TACs automatically.

Solution

The cloning of DEV DB schema referring to the respective TACs is not a good practice. Talend Support does not recommend having such dynamic synchronizing of the TAC DB reflected in other environments, especially in the production environment.

The only recommended way to do this is to manually create users. If the environment is being set up for the first time, you can clone the dev TAC DB to test and prod. However, the practice of cloning it daily is also not recommended.

Dynamic synchronizing with the production environment may cause the corruption of the database. Instead of letting the prod environment get into an unknown state, Talend Support recommends that you manually maintain the entries within the TAC, as this holds the complete record of transactions to the DB.

Question

How can you protect production passwords, which can easily be seen by tContextDump?

Talend Studio is primarily a development and testing environment. Developers should be given only contextual information related to the development and testing environment; they should not have access to production passwords.

It is OK if a developer tries to print test/dev passwords; production remains secure. However, a developer could still add malicious code to a Job to reveal a production password, or send email to somebody from within a Studio Job.

Answer

This type of code tampering can be prevented using a code review. When a Job has been tested, reviewed, and is ready to be deployed, a qualified individual can set up the contextual information as parameters in Talend Administration Center (TAC), which is password restricted.

Problem Description

A Job using a tWaitForFile component waits for the files to arrive in its input/scan directory. However, sometimes when a file is copied/created into the input directory, the tWaitForFile component triggers too soon, and the next component executes prematurely.

Root Cause

This can occur with large files, particularly in Linux environments.

Solution

1. Double-click the tWaitForFile component.

2. Click the Advanced Settings tab.

3. Select the Wait for file to be released check box, and set the Every (in ms) parameter so that the next component won’t trigger until after that file operation is complete.

For files over 2 Gb, set this parameter to at least 1,000.

Problem Description

Conversion of Oracle SCD to Oracle ELT SCD isn't working. The tables previously loaded using the tOracleSCD component are not updating when using the tOracleSCDELT component.

Root Cause

This issue is due to the change in the default date formats. The change is not picked up by tOracleELTSCD component, and the table update doesn’t occur.

Format of Oracle SCD: DD-MM-YYYY 12:00:00

Format of Oracle ELT SCD: DD-MM-YYYY 00:00:00

Solution

Before using the tOracleSCDELT component on a table using a tOracleSCD component, modify the END_DATE column using the TRUNC function.

Update the END_DATE column in the table being implemented for SCD in the database, where Table_name is the target table that needs to be updated, as shown in the following SQL query:

UPDATE table_name set END_DATE=trunc(END_DATE)

This allows the updates to occur as expected.