Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Have a redshift output component pointing at a table with 37 million rows in it, being fed by a tMap and several lookups. When I start the job it doesn't even try to run the input query or the tMap, it just loads the lookup tables and then says "starting" above the flow to the tRedshiftOutput component.
This issue has been occurring all over the place across all my jobs, they just refuse to run.
Anyone ever heard of this or know what my problem might be?
Image is "Live Statistics" from TAC.
Hi,
It seems the DB you were connecting was unresponsive. Did you try to restart your DB to see if it work?
Is it OK with you that output your data into a file instead of DB?
Best regards
Sabrina
And here is an image showing the current status:
Maybe check the checkbox "Die on error" on the tRedshiftOutput component and run it to see what is causing it to hang.
I put a system.out.println in the input query box and it won't even run that. No idea what it is doing.
Are you sure you have access from wherever you are running the job to the redshift table?
Hi,
It seems the DB you were connecting was unresponsive. Did you try to restart your DB to see if it work?
Is it OK with you that output your data into a file instead of DB?
Best regards
Sabrina
Writing to a file works.
It seems to be a problem with the database.
Thank you for the help.
I have exactly the same issue. Was this ever resolved?
This was almost a year ago now!
I can't really say what exactly it was, all I've got is a few key things we've done since:
- Use tRedshiftOutputBulkExec instead of a standard tRedshiftOutput. This will write files to S3 and then use the COPY command to load the data into redshift.
- Make sure to use streaming in the mysqlInput advanced options (for large result sets).
- We moved away from tMap in favour of tJavaFlex and doing all the mapping programmatically where feasible.
Let me know if you are still stuck after trying some of this stuff and I'll see if I can think of anything else.