[resolved] No way to dynamically override dropdown component settings via parameters

[dan2]

Edited the title to help others in the future and marked the comment with the workaround.

Please see the screenshots below. I have a Joblet triggered by Component Ok in the main Job. The first step in the Joblet sets the parameters dynamically for a tS3Connection. The Debug log shows that the tS3Connection component is failing to read the new context. It should have updated to PROFILE_INHERIT_CREDENTIALS, but instead stays with the default even though the context was definitely updated by the tJava. How can I correct this?

 

 

 

 

I went ahead and output the full context upon failure of the tS3Connection component and can show that the context was updated, so tS3Connection seems to not be reading it. I also tried adding all variables as globals, but this did not work either.

[dan2]

@Richard Hall​ You can reproduce the bug by setting  __CREDENTIAL_PROVIDER__ = "PROFILE_INHERIT_CREDENTIALS" in the Job context, then calling a joblet with tS3Connection in it with Credential Provider option set to "Inherit credentials from AWS". Then run in debug and tS3Connection_1 - Parameters: will not show the override. Not sure how that is a timing issue.

[dan2]

@Richard Hall​ I've attached a scaled-down version reproducing the bug. I simply added the one override for __CREDENTIAL_PROVIDER__ and when you run the debug, it does not pick it up. To be clear, "PROFILE_INHERIT_CREDENTIALS" and "INHERIT_CREDENTIALS" are two different options. It continues to use "INHERIT_CREDENTIALS" per the drop-down, instead of the override from the Job Contexts.

[Anonymous]

Hang on, I missed this. You cannot use Context Variables to set a drop down option in a component. This value will be compiled into the job. Sorry, I was focussing on Context variables and missed what you were trying to set.

[dan2]

How can I dynamically set the component's dropdown? Different options are needed for Dev/QA/Prod environments.

[Anonymous]

There is a way you can attempt this, but I've not actually done this with tS3Connection components. I HAVE done this with DB components where drop downs needed to be used and I needed to dynamically choose between them.

Essentially the process was this.

1. Create all of the connection components you may need and link them (via a RunIf) to a "dummy" tJava at the start of your job. Depending on your environment, only one of the paths will be followed.
2. After the connection components, add a tJava component. Use an OnSubJobOK link from your "dummy" tJava
3. Choose the connection component name that you will use for all of your S3 components throughout the job. I usually use the "_1" connection.
4. In the tJava that you have added, you need to assign the connection you have used to the globalMap replacing the object held by default in the code (take a look at the Talend code for this).

As an example.....

Let's say you have 3 tS3Connection components called tS3Connection_1, tS3Connection_2 and tS3Connection3.

Your RunIf links will instantiate the one that you want at the beginning of the job. You now need to set that connection to the globalMap as the tS3Connection_1 connection.

This is done like below....

if({condition to identify connection_2}){

globalMap.put("conn_tS3Connection_1", conn_tS3Connection_2);

}else if({condition to identify connection_3}){

globalMap.put("conn_tS3Connection_1", conn_tS3Connection_3);

}

As long as this takes place before any of your components using the connection, it should enable this.

[dan2]

Thanks for this workaround. That makes sense conceptually. I'll give it a try and report back. This is definitely a feature request. It would be much nicer if you can override dropdown selects like you can with all other component text inputs.

[Anonymous]

I believe the reason for this way of working is that different Java code is required for each option selected. This means it is hard to parameterise it without adding further complexity. Along with this, there is the consideration that in many (DEV, UAT, PROD, etc) environments, the "type" of connection will be the same across the board. I do understand that is not the case (I have experienced it), but commonly it is the case.

One of the key advantages of Talend (a reason why I have worked with it for over a decade before I joined the company) is that a bit of Java can open many doors. So while this isn't "out of the box" functionality, you can build it if you need it.

But I will speak to my colleagues in R&D to see if this is possible.

Sorry for the initial confusion in what you were asking.

[dan2]

To elaborate on the use-case, AWS has Roles assigned for access to Buckets and other objects. For security purposes, we cannot assume roles from our local machines and have more limited users for development that access via local Profile. The expectation of PROFILE_INHERIT_CREDENTIALS option under __CREDENTIAL_PROVIDER__  in tS3Connection is that it might meet both needs with some form of try/catch/try, but that does not appear to be the case, so when the job is promoted to Talend Cloud, it needs INHERIT_CREDENTIALS and we set the Role to be assumed. In general, any input of a component, should be able to be dynamically set. It would also be useful to be able to instantiate a component using wrapper functions in tJava, especially reusable connections. Granted I can rewrite simple S3 tasks as my own Java Routines, but being able to use that connection with another component like those used to load data from S3 to Redshift would be valuable. Otherwise I'm forced to rewrite anything requiring S3 as Java. In this case, it looks like the proposed workaround will hopefully solve the problem, but I can image other iterative use-cases where not being able to fully parameterize will be far more impractical.

[Anonymous]

I understand. I really do. I will raise this. However, as I said, it is not always easy to parameterise in all situations. There is also a balance between ease of use for most use cases (for people who don't have a computer science degree) and providing something that can be extended by those who are a little bit better equipped. It is a fine line and I'm not going to claim that it is always perfected.

I'd be interested to see how you get on with this workaround. If you could give a "yay" or a "nay" to whether this worked, I'd really appreciate it.

[dan2]

Unfortunately, globalMap.put("conn_GetSecretsAndDefaults_1_tS3Connection_2", "conn_GetSecretsAndDefaults_1_tS3Connection_1") does not work and throws a null pointer exception because conn_GetSecretsAndDefaults_1_tS3Connection_2 never gets instantiated since the RunIf only creates conn_GetSecretsAndDefaults_1_tS3Connection_1. Any idea how to remedy this situation?