topic Re: [resolved] No way to dynamically override dropdown component settings via parameters in Talend Studio

[resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Fri, 15 Nov 2024 23:07:29 GMT

Edited the title to help others in the future and marked the comment with the workaround.

Please see the screenshots below. I have a Joblet triggered by Component Ok in the main Job. The first step in the Joblet sets the parameters dynamically for a tS3Connection. The Debug log shows that the tS3Connection component is failing to read the new context. It should have updated to PROFILE_INHERIT_CREDENTIALS, but instead stays with the default even though the context was definitely updated by the tJava. How can I correct this?

I went ahead and output the full context upon failure of the tS3Connection component and can show that the context was updated, so tS3Connection seems to not be reading it. I also tried adding all variables as globals, but this did not work either.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

Anonymous — Sun, 13 Mar 2022 02:18:51 GMT

This likely a timing issue. Can you show a screenshot of your job and a separate screenshot of your joblet that sets these values?

A joblet is essentially a collection of components that act is if they are part of the job you have used it in. My suspicion here is that your tS3Connection is instantiated before the context variables are set.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 03:21:33 GMT

Hi @Richard Hall:

GetSecretsAndDefaults_t Joblet is the 2nd step to run in the PreJob:

Step 1Step 2 (Joblet GetSecretsAndDefaults)

Step 3 (After TRIGGER_INPUT_1 is tJava3 which finds the user is not the Cloud user and sets the Local parameters via context variable changes)

Step 4 ( some unrelated setting of globals)

Step 5 (tS3Connection_1 <-- the issue. The parameter overrides are not getting picked up by this Component. I have also tried to set the parameters as global, but that did not work.)

Step 6 (on failure, we are able to see that the context variables were updated, but do not seem to have been picked up by tS3Connection_1)

Here are some larger pictures

Thanks again for your help!

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 19:09:45 GMT

@Richard Hall As a test, I also attempted to use tRowGenerater->tJavaRow->tContextLoad->tJava->tS3Connection. It loads the context variables ahead of the tS3Connection, but it still does not register the new parameter value. This feels like a bug..

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 21:18:29 GMT

@Richard Hall I went ahead and defined the Contexts directly in the Joblet to rule out an issue there and found that that did not even work. Oddly enough, printing the context immediately upon Joblet start comes back totally empty. I only am able to see the Context if I run context.synchronizeContext();, but that seems to only result in it being available in the prior tJava and tS3Connection is not affected. I believe tS3Connection sees no context from the beginning and does not care that a synchronize command occurred. To align with the documentation, I went ahead and put what I needed into a Context Group with the various context environments. I then dragged that Context Group into the Joblet. Talend appears to totally ignore the values and show blank context when the Joblet begins.

I then removed all context from the joblet and the main job and dragged the Context Group into the main job. The contexts still show up blank when I run it this way and the overrides are still not working.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

Anonymous — Sun, 13 Mar 2022 21:30:55 GMT

Can you try this? After you have set your context variables in your tJava (straight after your Trigger_Input), use an OnSubJobOK link to your next component. I still think this is a timing issue. You connection component would be better placed in a subsequent SubJob.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 22:00:41 GMT

@Richard Hall That unfortunately did not change the outcome. I'm beginning to think that the variables attached to the parameter in tS3Connection cannot be overridden due to a Talend code bug. Ideally, I do like the Context Group approach, but even hard-coding __CREDENTIAL_PROVIDER__ in the main job is not overriding tS3Connection.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 22:07:43 GMT

Am I misunderstanding the meaning of "The variable attached to this parameter is:"__CREDENTIAL_PROVIDER__"? Am I not allowed to override component variables using environment context? If the settings cannot be changed dynamically in this manner, than I'd want to understand the best practice running the job across different environments that have different connection configs.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

Anonymous — Sun, 13 Mar 2022 22:15:34 GMT

Context variables shouldn't really be set within a running job, unless you are using a tContextLoad at the beginning of the job. However, I have used this technique quite a lot. Sometimes I have had to use this code to ensure the values are set immediately.....

context.setProperty("{context name case sensitive}", {context value})

....my next suggestion was to try that. The other thing to try is using a tContextLoad component.

But, when changing values in a Job, it is arguably better to use the globalMap. Try my suggestions above. If they don't work, then the globalMap should. It is very difficult to identify if this is a bug without having access to your job. But from what I have tried in v8, I don't think this is a bug. I still think this a timing issue.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 22:24:12 GMT

I took a step back and now am attempting this by adding the context per best practice, as a Context Group to the main Job, but that does not even work for this override. I have already attempted to set __CREDENTIAL_PROVIDER__ in the globalMap and using context.setPropertly. Same results. To be clear, __CREDENTIAL_PROVIDER__ is a dropdown in tS3Connection, so it's not like I can input java into the field to reference a global..

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 22:32:14 GMT

@Richard Hall You can reproduce the bug by setting __CREDENTIAL_PROVIDER__ = "PROFILE_INHERIT_CREDENTIALS" in the Job context, then calling a joblet with tS3Connection in it with Credential Provider option set to "Inherit credentials from AWS". Then run in debug and tS3Connection_1 - Parameters: will not show the override. Not sure how that is a timing issue.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 22:45:34 GMT

@Richard Hall I've attached a scaled-down version reproducing the bug. I simply added the one override for __CREDENTIAL_PROVIDER__ and when you run the debug, it does not pick it up. To be clear, "PROFILE_INHERIT_CREDENTIALS" and "INHERIT_CREDENTIALS" are two different options. It continues to use "INHERIT_CREDENTIALS" per the drop-down, instead of the override from the Job Contexts.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

Anonymous — Sun, 13 Mar 2022 22:51:55 GMT

Hang on, I missed this. You cannot use Context Variables to set a drop down option in a component. This value will be compiled into the job. Sorry, I was focussing on Context variables and missed what you were trying to set.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 22:57:54 GMT

How can I dynamically set the component's dropdown? Different options are needed for Dev/QA/Prod environments.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

Anonymous — Sun, 13 Mar 2022 23:31:57 GMT

There is a way you can attempt this, but I've not actually done this with tS3Connection components. I HAVE done this with DB components where drop downs needed to be used and I needed to dynamically choose between them.

Essentially the process was this.

Create all of the connection components you may need and link them (via a RunIf) to a "dummy" tJava at the start of your job. Depending on your environment, only one of the paths will be followed.
After the connection components, add a tJava component. Use an OnSubJobOK link from your "dummy" tJava
Choose the connection component name that you will use for all of your S3 components throughout the job. I usually use the "_1" connection.
In the tJava that you have added, you need to assign the connection you have used to the globalMap replacing the object held by default in the code (take a look at the Talend code for this).

As an example.....

Let's say you have 3 tS3Connection components called tS3Connection_1, tS3Connection_2 and tS3Connection3.

Your RunIf links will instantiate the one that you want at the beginning of the job. You now need to set that connection to the globalMap as the tS3Connection_1 connection.

This is done like below....

if({condition to identify connection_2}){

globalMap.put("conn_tS3Connection_1", conn_tS3Connection_2);

}else if({condition to identify connection_3}){

globalMap.put("conn_tS3Connection_1", conn_tS3Connection_3);

}

As long as this takes place before any of your components using the connection, it should enable this.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Sun, 13 Mar 2022 23:43:33 GMT

Thanks for this workaround. That makes sense conceptually. I'll give it a try and report back. This is definitely a feature request. It would be much nicer if you can override dropdown selects like you can with all other component text inputs.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

Anonymous — Sun, 13 Mar 2022 23:52:32 GMT

I believe the reason for this way of working is that different Java code is required for each option selected. This means it is hard to parameterise it without adding further complexity. Along with this, there is the consideration that in many (DEV, UAT, PROD, etc) environments, the "type" of connection will be the same across the board. I do understand that is not the case (I have experienced it), but commonly it is the case.

One of the key advantages of Talend (a reason why I have worked with it for over a decade before I joined the company) is that a bit of Java can open many doors. So while this isn't "out of the box" functionality, you can build it if you need it.

But I will speak to my colleagues in R&D to see if this is possible.

Sorry for the initial confusion in what you were asking.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Mon, 14 Mar 2022 00:16:55 GMT

To elaborate on the use-case, AWS has Roles assigned for access to Buckets and other objects. For security purposes, we cannot assume roles from our local machines and have more limited users for development that access via local Profile. The expectation of PROFILE_INHERIT_CREDENTIALS option under __CREDENTIAL_PROVIDER__ in tS3Connection is that it might meet both needs with some form of try/catch/try, but that does not appear to be the case, so when the job is promoted to Talend Cloud, it needs INHERIT_CREDENTIALS and we set the Role to be assumed. In general, any input of a component, should be able to be dynamically set. It would also be useful to be able to instantiate a component using wrapper functions in tJava, especially reusable connections. Granted I can rewrite simple S3 tasks as my own Java Routines, but being able to use that connection with another component like those used to load data from S3 to Redshift would be valuable. Otherwise I'm forced to rewrite anything requiring S3 as Java. In this case, it looks like the proposed workaround will hopefully solve the problem, but I can image other iterative use-cases where not being able to fully parameterize will be far more impractical.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

Anonymous — Mon, 14 Mar 2022 00:28:09 GMT

I understand. I really do. I will raise this. However, as I said, it is not always easy to parameterise in all situations. There is also a balance between ease of use for most use cases (for people who don't have a computer science degree) and providing something that can be extended by those who are a little bit better equipped. It is a fine line and I'm not going to claim that it is always perfected.

I'd be interested to see how you get on with this workaround. If you could give a "yay" or a "nay" to whether this worked, I'd really appreciate it.

Re: [resolved] No way to dynamically override dropdown component settings via parameters

dan2 — Mon, 14 Mar 2022 04:06:12 GMT

Unfortunately, globalMap.put("conn_GetSecretsAndDefaults_1_tS3Connection_2", "conn_GetSecretsAndDefaults_1_tS3Connection_1") does not work and throws a null pointer exception because conn_GetSecretsAndDefaults_1_tS3Connection_2 never gets instantiated since the RunIf only creates conn_GetSecretsAndDefaults_1_tS3Connection_1. Any idea how to remedy this situation?