Talend job hanging "starting"

[GarethPenrose]

Have a redshift output component pointing at a table with 37 million rows in it, being fed by a tMap and several lookups. When I start the job it doesn't even try to run the input query or the tMap, it just loads the lookup tables and then says "starting" above the flow to the tRedshiftOutput component.

This issue has been occurring all over the place across all my jobs, they just refuse to run.

Anyone ever heard of this or know what my problem might be?

Image is "Live Statistics" from TAC.

[GarethPenrose]

And here is an image showing the current status:

[Anonymous]

Maybe check the checkbox "Die on error" on the tRedshiftOutput component and run it to see what is causing it to hang.

[GarethPenrose]

Thank you for the suggestion. I gave it a go but no luck, it seems there is not an error because the component does not die and I do not get a message in the log, it still hangs.

[GarethPenrose]

I put a system.out.println in the input query box and it won't even run that. No idea what it is doing.

[Anonymous]

Are you sure you have access from wherever you are running the job  to the redshift table?

[Anonymous · Accepted solution]

Hi,

It seems the DB you were connecting was unresponsive. Did you try to restart your DB to see if it work?

Is it OK with you that output your data into a file instead of DB?

Best regards

Sabrina

[GarethPenrose]

Writing to a file works.

It seems to be a problem with the database.

Thank you for the help.

[Anonymous]

I have exactly the same issue. Was this ever resolved?

[GarethPenrose]

This was almost a year ago now!

I can't really say what exactly it was, all I've got is a few key things we've done since:

- Use tRedshiftOutputBulkExec instead of a standard tRedshiftOutput. This will write files to S3 and then use the COPY command to load the data into redshift.

- Make sure to use streaming in the mysqlInput advanced options (for large result sets).

- We moved away from tMap in favour of tJavaFlex and doing all the mapping programmatically where feasible.

Let me know if you are still stuck after trying some of this stuff and I'll see if I can think of anything else.