topic Re: Talend Studio feedback for Talend internal dev team in Talend Studio

Talend Studio feedback for Talend internal dev team

dan2 — Fri, 15 Nov 2024 23:58:08 GMT

We are starting our adoption of Talend and, in general, it is much more robust that our current Informatica software, however there is some room for improvement. After working through the creation of my first production job, I have the following constructive feedback regarding Talend Studio:

Does not let you auto-organize job components
Workflow lines between components cannot be moved directly, so overlap components when you continue underneath a row of components/subjobs. The only solution I have found is to make the last component in line misaligned with the rest on the same row.
Cannot automatically zoom to fit all components on screen
Components can get stuck in ways where you cannot double-click them
Subjob bubbles cannot easily be resized manually
The red error box around component icons is very hard to click into for error details most of the time
The Code Viewer and Code editor should just be the same thing. It was initially confusing to figure out why line numbers would not show in code viewer after making the settings change
Direct searched for components in the Palette by exact name do not show the component on the top and still can have it hidden in an unexpanded hierarchy
Items are not in alpahbetical order in certain areas of the Repository heirarchy, such as under Metadata. For example, it always takes me a while to find Generic Schemas.
If I try to change the case of a Context Variable, it won't let me until I fully delete the variable since the uniqueness test is triggered improperly in a self-referential manner.
When I retrieve schema on an existing Table schema after there were updates on the database-side, it sometimes says that there were no updates, even afer it does in fact update the schema.
There is no consistency in product offering naming conventions. Even though I am using what is considered Talend Studio. The title at the top of my screen is Talend Cloud Real-Time Big Data Platform (7.3.1.20200219_1130). If I go into Help>About Talend Studio, it also says that it's Talend Cloud. If I download Data Studio while logged in, I get something different than not logged in. It is very confusing.
Regarding the Talend Help Center ribbon, which looks like a browser advertisement bar, it is not apparent how it can be removed. There are almost no components available in the Exchange. Videos does not got to Talend Academy or a full content provider and only shows four videos.
Components do not display a help icon linking to their documentation page.
Many of the Cheat Sheets under help are missing and lead to "Cheat sheet content file * not found."
If you maximize a panel, then minimize it, it fully vanishes. Then, for example, you need to double-click into a Component in the designer to get it back, which will have the panel show up in an obscure place, then clicking maximize again will put the panel back in the correct place.
Clicking Debug Run tab under Run freezes the application for several minutes.
Project files will not import unless you start from the initial splash screen.
I should not need multiple logins to go to the Talend forum and other areas of the Talend website. That was very confusing.
tDBBulkExec has no database Schema override, so the Schema option set in tDBConnection is the only one that can be targeted, so multiple connection are needed to target different schemas.
Filter cannot be used to route multiple output rows. For example, one file has more than one report, identified by the first column of the row. Rows need to be routed to different paths. This does not appear to be easily achievable.
There is a Prejob and Postjob, but no component that I can use for the parallel flows that occur in the main job, so I am forced to use something like tSleep and label it Start_Main_Job_1 in order to make it look nice.
YOUR FORUM DOES NOT HAVE A FEEDBACK AND FEATURE SUGGESTION TOPIC....

Re: Talend Studio feedback for Talend internal dev team

dan2 — Sat, 10 Jul 2021 18:54:31 GMT

24. If you copy and paste components instead of showing up near your cursor or near the copied component, it seemingly shows up in a random potentially off-screen location and needs to be tracked down.

25. If you copy a group of components with a tMap in the middle and change the schemas on either side and resync etc., no matter what, the tMap will not let go of the older columns and simply needs to be deleted and recreated.

26. There is no notification component and instead a warning component must be used instead to signify the end of the job and pass an exit/return code. Warning should not be the standard for this. Something like tUpdateJobStatus or tExitCode should exist.

27. Running steps should be highlighted. It is hard to tell which step is currently processing when testing.

Re: Talend Studio feedback for Talend internal dev team

gjeremy1617088143 — Mon, 12 Jul 2021 07:34:23 GMT

21 . you could use filter expression of tMap multiple outputs.

14.just presse f1 on a field of a component it will redirect you to the documentation of the component.

22.If you use studio version : tParallelize or tPartitioner else you have other way to make parallelization in open studio.

26.there is an internal variable ExitCode also you can write directly in the console with just few line of java.

3.in the Outline view you can see a full view of your job and navigate more easily in.

28.Like every tool Talend have limitations but as far i know it's one of the most powerfull and robust. Just keep to dig in you 'll see you could do a lot of thing with.

Re: Talend Studio feedback for Talend internal dev team

Anonymous — Mon, 12 Jul 2021 11:10:22 GMT

Hi @Not defined Not defined,

Thanks for raising this feedback. There is some very interesting content in there. I COMPLETELY understand some of the points you raise (the first 4 being frustrations I have had in the past). There are also a couple of points where I feel I can offer advice or a workaround. Naturally, I will pass all of this on to our R&D team, but let me see if I can help with some of the issues....

For number 6 you will get far more info from the Code tab (bottom left of the developer canvass). If you ever get errors with components which show up as a red box, this is the first place you should look. They will more than likely be Java compilation errors caused by parameters of components not having been filled in properly. It is always a good idea to look at the code tab and check for red blocks on the right to identify potential config errors before you try to run/build as well.

I'm not sure about point 7, but let me try to explain how to resolve the line numbers frustration. When you click on your code tab, you will see the code that has been autogenerated by the placement and configuration of components. You cannot edit this code, it is just so that you can see what is happening. Sometimes you may notice that the line numbers are not present. To switch these on, right click on the thin grey line down the left side of code screen and you will see a box like this....

Simply select "Show line numbers" and they will appear.

For number 8, I can see what you mean. This doesn't work elegantly, but it does work if you click on the magnifying glass.

But there is a MUCH better way of finding a component that you may need, which is why I do not worry about the issue you raised here anymore (I used to). If you start typing the name of the component you want to use on the developer canvass, it will do exactly as you wish.....

Number 10 IS a pain. However, here is a workaround. If you have a Context named "BOB" and you want it changed to "bob", you can do this. First rename it to "BO" by deleting the final "B". Then overwrite it again with "bob". This will work. Yes, it is a pain, but this will work for you. As I said, I will raise this though.

For number 11, this can happen if you change the schema to be built-in. Also, with repository schemas you can find that the columns are made available, but are not switched on automatically. In some cases you may not want your job to automatically add columns which are switched on. This one can catch you out, I know.

I have to say I like your suggestion for 14. But if you click on the component and hit F1, it will load the help page you require. It's a bit like an Easter Egg 🙂 But I understand how this is not easy to find.

If I understand 16 correctly, yes, that can be a frustrating experience. There is a quick fix to this....

I'm not sure I understand 18. Can you elaborate on that for me? It doesn't ring any bells for me.

For 19, we are working on that as we speak. Multiple logins have built up during the lifespan of Talend and we are very aware of the issues caused. Since Talend has grown via organic growth and acquisition, it has been something that has happened without intention. As I said, this will be fixed.

I may be misunderstanding 20, but I don't see that issue. This might be because I am doing something different to what you are explaining, but let me show you my Snowflake version of this....

For 21, @guenneguez jeremy explains this, a tMap is what you want for this. There is a lot of crossover between a tMap and a filter. I tend to use the tMap in the majority of cases. The example you gave is perfectly handled by the tMap.

For 22, @guenneguez jeremy has also nailed this. I should add that the tPreJob and tPostJob components have very important uses. The tPreJob will fire before ANY of the rest of the job (even context variable settings) fire. So you can handle things like dynamic contexts there. The tPostJob is MUCH more significant. Sometimes when you are running a job it may fail due to an unexpected nullpointerexception or something that causes another Java runtime error. Without the tPostJob the job would just fail and end. With a tPostJob the job will fail, but the tPostJob will ALWAYS be fired. Therefore you can use it to clean up and/or catch errors. It's incredibly useful.

Number 23 is my fault. We used to have a section in the Community for this before we migrated to this platform. Unfortunately it was seldom used for the right purpose at all. We also realised that we were asking in two separate areas for Feature Suggestions. You can raise these in Jira (https://jira.talendforge.org/). First, you will need another login at the moment and you can find a page to register here (https://login.talend.com/login.php). Once you've registered you will need to login via the Jira URL I gave previously. I know, this is really not ideal to have to do this, but this WILL be fixed. It is just a matter of time I promise.

I couldn't recall having experienced 25, so I have tried it out according to your description. I *think* that maybe you are not updating the schema by pushing it back. These screenshots show what is required....

So, I deleted half of the components on the right, then clicked the double arrows pointing back (in the red box). This leaves you with this....

If you check your tMap, you will see that the columns have bee updated.

As I said, I am open to being shown I have misunderstood. I am also using a Mac and I know that sometimes there are differences between Mac and Windows versions (although I've only seen things struggling on Mac after a new Mac OS or update).

For 26 there are a plethora of ways to indicate the end of a job. As mentioned before, the tPostJob component is really useful for enabling you to ALWAYS call the end of a Job and to allow you to do whatever you like to indicate the end of a Job. I have used emails, database updates, SMS messages, Slack messages, .....I've even enabled my Google Home device to tell me a Job has completed in the past. One of the main benefits of Talend for me is that I can use ANY third party API with it. All I need is a bit of Java and an understanding of the API.

For running steps you can build these in with several components (tJava, tJavaFlex, tLogRow, etc), there is also Log4J. This tutorial/article might give you some pointers here....

https://community.talend.com/s/article/Log-j-tips-and-tricks-I8730

As I said, thanks so much for raising these points with us. While I have given a few workarounds, that is not to say that we do not need to look further into these areas. I will now be in touch with several other teams to highlight these points.

Regards

Richard

Re: Talend Studio feedback for Talend internal dev team

Anonymous — Mon, 12 Jul 2021 14:32:21 GMT

Hi @Not defined Not defined ,

Thanks for sharing your feedback.

For 13, I'm not sure I see what ribbon you're talking about; Feel free to paste a screenshot here or open a DOCT issue in JIRA, for us to look into this.

For 14., not intuitive but a workaround, press F1 on any component icon on the canvas should take you to the corresponding documentation (online). But definitely a good old question mark would make it more obvious.

For 15, we'll look into the cheat sheets issues (was this in the Data Profiling perspective? ).

For 19, we have an initiative to simplify the authentication throughout the whole Talend ecosystem. As all what touches sign-in and identity management, this is not a simple topic. But we hear you.

Thanks again for the feedback.

Elisa

Head of Documentation & i18n

Re: Talend Studio feedback for Talend internal dev team

dan2 — Mon, 12 Jul 2021 14:43:54 GMT

Thanks! These were all very helpful. #14 and #3 are going to save me a lot of time going forward. I just wish that there were more options to better organize the Designer objects automatically like Informatica has. A lot of time was wasted to trying to get lines to not overlap object and the things to be organized cleanly.

Re: Talend Studio feedback for Talend internal dev team

gjeremy1617088143 — Mon, 12 Jul 2021 14:48:47 GMT

for designer, you'll see with the time you can organize thing pretty clearly, could you send an example of line overlap object , i'm very curious about that

cause you can organize the layers of subjob and components for example you can right click on an empty area of a subjob and then you have options like bring to front or bring to back

in the palette you can also right click on component you use a lot and set it as a favorite,

it will be shown in the first folder of the palette

Re: Talend Studio feedback for Talend internal dev team

dan2 — Mon, 12 Jul 2021 15:19:32 GMT

Hi @Elisabeth Sabot ,

Thanks for your response.

13. Before submitting a bug report. I am simply trying to hide the following toolbar. It looks like it will only let me move it:

If you go to Exchange, there is only a very limited list of about 26 components. There is no next page button. Is this accurate?

15. I'm in the Integration perspective. Here's what happens:

Please also note, this forum allows me to copy and paste images from the Windows clipboard into the editor, such as through the Snipping Tool, but then it drops those images when submitted. Very frustrating. I had to go back in and save each image as a file and upload then one-by-one.

Re: Talend Studio feedback for Talend internal dev team

dan2 — Mon, 12 Jul 2021 15:32:43 GMT

I'd like to be able to drag the line itself around like I would in a tool like Vizio

vs. having to move the last object in the line a little down.

Re: Talend Studio feedback for Talend internal dev team

gjeremy1617088143 — Mon, 12 Jul 2021 15:57:00 GMT

you don't need to use an onsubjob ok link after the component wich finish the prejob line, the tprejob line will be executed before everything else (it's it purpose)

after this in general when you work with subjob you will use the on subjob ok link between the first component of each subjob (it was designed to avoid z pattern).

Between simple components like dbconnection (use on component ok instead of on subjob ok).

Send me Love and kudos

Re: Talend Studio feedback for Talend internal dev team

dan2 — Mon, 12 Jul 2021 16:24:26 GMT

Thanks for these thorough explanations. This is a big help and it's nice to see that the dev team is engaged with the user community.

6. Some errors do allow you to click them from the designer and get a little blurb. Agreed that the Code tab is better for this. Double-clicking the error box should simply open the offending line in the code editor.

7. There is a View called Code Viewer, which is separate from the Code tab. I ultimately removed it, but it is worth mentioning that line numbers cannot be added to it and it seems a bit redundant.

8. Even when clicking the magnifying glass or hitting enter, there are certain items that it found that are hidden under their herierarcy. It should automatically expand all related entries when doing a search. Typing into the designer is a neat trick and I'll definitely start using that more.

10. I'd go as far as to call this a bug.

14. Definitely a bit of an Easter Egg. Seems like adding that question mark icon in various places would improve user adoption of the tool. Added bonus if when you hover over the question make, the tooltip hints at the F1 hot-key. Please note, I am comparing this tool to Informatica, which has these bells and whistles.

20. This is more about Schema as a database target vs it's structure. For example, I have a database/schema named fin_staging and it has a table in it called test_table, so the full path to the table is fin_staging.test_table. In the tDBConnection, it has be specify that Schema option to fin_staging. If I do not set it, anything that I do in tDBBulkExec will default to the Redshift "public" database/schema. If I set Table Name in tDBBulkExec to "fin_staging.test_table" and have by tDBConnection set to "xyz", it creates a table named "xyz.fin_staging.test_table" and does not override "xyz" in favor of "fin_staging". If I do not set Schema in the tDBConnection, it will default to "public.fin_staging.test_table". I'm trying to reuse the same connection, but am not able to when targeting multiple database schemas for this reason.

21. I think that I need to test this a bit more. If I'm understanding correctly, I would hit the plus for an additional output and can put the other row schema in since it differs in columns and data types. Then, I'd use an expression filter like row3.record_type="RACT0010"? The tFileInputDelimited requires a defined schema as well, so do I just use a more generic schema to read in the file like one that assumes the maximum amount of potentially available columns and defaults them all to varchar(255)?

22. With the pre-job, I have seen the behavior that the rest of the job will continue on, even if some of the tDBRow steps fail. Do I avoid this by adding tDie to them with Priority "Fatal"? I am seeing a growing trend where people are simply chaining their main job to the Prejob since it avoids the issue that I mention and looks a bit cleaner than having the main job component chain look orphaned, not having a nice looking starting icon like the Prejob and Postjob. Is the best practice for the main job chain/threads to be added with a tParallelize or tPartitioner starter component, even if there is just one main chain/thread?

25. As time permits, I'll try to recreate the issue and test your solution. For my use-case, I might not needs so many mappings anyway provided that the solution for 21 works.

26. Similar to 22, it would be nice to have a component with an icon that signifies the end of the main job, Not necessarily the Postjob, which I use to just close my database and S3 connections, even if there are failures. I'll definitely take a look at the linked tutorial when I can.

I ran into an additional issue yesterday worth mentioning. When establishing my Redshift tDBConnections and then re-using them throughout the job. We have our database seemingly configured aggressively to boot connection that stay idle for 60 seconds, which kills the established connections. The Redshift components don't know what to do when it is their time to run and there is no option to tell them to attempt re-connect. The job gets stuck in an endless loop of saying that the connection was booted and does not throw an error. I had to add Additional parameters: tcpKeepAlive=true&TCPKeepAliveMinutes=1 in order to resolve the issue. It would be nice if some of these pitfall parameters covering the timeout scenarios were standard options when configuring the Redshift connection. When browsing the documentation, it seems that they may have been at one point, but are not there anymore. It took me several hours to solve that issue since I really wanted to use the same connection and commit all the changes at the end of the job.

Thanks,

Dan

Re: Talend Studio feedback for Talend internal dev team

dan2 — Mon, 12 Jul 2021 16:33:29 GMT

I linked the Prejob to the main job in order to ensure that any errors in the Prejob would not allow for the main job to run. Was experiencing tDBRow errors where the main job still ran.
The above screenshots were mostly to demonstrate the overlap issues with the OnSubjobOk lines regardless of its usage. If I wanted to simply continue the Prejob onto another grid row, I'd have to lower the last component on the top instead of simply being able to modify the shape of the line directly like you would in a charting tool.

Re: Talend Studio feedback for Talend internal dev team

gjeremy1617088143 — Tue, 13 Jul 2021 07:07:17 GMT

in the configuration of the studio you can chose line instead of curve for the link design, personnaly i prefer line it's clearer.

Re: Talend Studio feedback for Talend internal dev team

gjeremy1617088143 — Tue, 13 Jul 2021 08:30:55 GMT

hope this trick can help you for line overlaping problems

Re: Talend Studio feedback for Talend internal dev team

Anonymous — Tue, 13 Jul 2021 16:50:11 GMT

Thanks Dan (@Not defined Not defined ) for your further explanations. I have got our product managers taking a look at your list of issues, so this elaboration certainly helps.

20) Oh I see what you mean. I tend to be less concerned with the number of connections I am using unless I am wanting to achieve an atomic transaction across multiple locations. Is there a reason why a connection per schema would be a deal breaker here? However, I do see that this would be frustrating, especially when your login can access multiple schemas simultaneously.

21) Yes, just hit + and add as many outputs as you wish. The row structure or schema can be as you wish for each output. If your file has a different schema for each row, it can get tricky, but you sound like you have the right idea to start with.

22) The tPreJob won't stop the main flow (not attached to the tPreJob or tPostJob) from starting unless there is a major issue in the tPreJob. However, you can control this with a bit of logic at the beginning of the main flow if you like. I suspect this could be handled with a tJava and some RunIf connectors. It's tricky to give you a solution without seeing the problem. But there will be a simple fix to this.

It is only the tPostJob that has guarantees on running.

Your issue with your connection closing and your suggestion seems reasonable. As an alternative, it might be a solution to use the connection in your Redshift component that is selecting/inserting/updating/deleting. This will only be triggered when you have something to do with the DB, so you are less likely to see that timeout. If you are opening a connection and then doing some potentially lengthy calculations over hundreds of thousands/millions of rows, your connection may be dropped in that time. Just a suggestion for another way of tackling this 🙂

Re: Talend Studio feedback for Talend internal dev team

dan2 — Tue, 13 Jul 2021 18:08:53 GMT

20) I prefer to commit all database changes at the end of the job after ensuring that everything staged properly. Spreading those changes across two or more sessions makes it a bit more complicated to manage multiple commits.

22) If I use the tJava approach to learning whether all Prejob items succeeded instead of just linking to the main flow via a chain of OnSubJobOk's, is there a status variable associated with the overall successful completion of the Prejob? Otherwise, it sounds like a lot of extra work to just eliminate a linking line.

Regarding the timeouts, that hopefully won't occur while a connection is actively being used. Per the earlier comment about commits, I'd rather use a single session for the whole job and commit at the end vs. opening a new session for each component step in the flow. This also makes it easier for our DBA's to potentially anticipate/manage resource availability for our process since they can look at the trending of a single session instead of many.

Re: Talend Studio feedback for Talend internal dev team

Anonymous — Wed, 14 Jul 2021 19:40:02 GMT

I came from an Informatica background (a LONG time ago) and I had to change the way that I thought about a mapping (that expression takes me back somewhat). With Talend, you have A LOT more freedom to create, but that does come with some extra work I guess. However, it doesn't take long to get into that way of thinking and that is when you see the doors it opens for you.

For example, you can do anything you can in Java using Talend (apart from GUI stuff of course....unless you try REALLY hard). With regard to 22, this is where you see this. As part of every Talend Job there is a HashMap called globalMap. This can be used similarly to how "global variables" are used in Informatica. So, if you have a way of assessing the success or failure of your tPreJob (or any other SubJob, series of SubJobs...pretty much anything), you can assign the status of that to the globalMap like so (assuming the value is true/false here).....

globalMap.put("tPreJobSuccess", true)

If you want to check it later in the job, you can access it like this (again, this is a boolean so the value stored as an object needs to be cast)....

((Boolean)globalMap.get("tPreJobSuccess"))

Regarding your reasoning for using single commits, that's fair enough. I was just curious.

Re: Talend Studio feedback for Talend internal dev team

qiongli — Wed, 08 Sep 2021 09:39:55 GMT

Hi @Not defined Not defined ,

About '15.Many of the Cheat Sheets under help are missing and lead to "Cheat sheet content file * not found."'

Which Cheat Sheets are missing? I test it on release 7.3.1 with License 'Talend Cloud Real-Time Big Data Platform'. The cheat sheets are as follow and no error like "Cheat sheet content file * not found." :