After executing for more than one hour, tasks are failing with the error shown below:
java.sql.SQLException: [Amazon](500150) Error setting/closing connection: Not Connected
This generally is caused by not having the proper security group configuration in Redshift (for example, whitelist the IP address of the server running the Spark driver).
Resolution
Get the hostname and/or IP address from the Remote Engine machine that is running the failed task.
Sign in to the AWS Console.
Go into the EC2 Management Console.
On the left navigation pane, look for the Network & Security header and click Security Groups.
Click the row with Default in the Group Name's column.
Select the Inbound tab, then click Edit.
Add Rule: Type=Redshift, Source=MyIP. (auto-fill)
Save it.
After making the changes, use the telnet command to verify whether the connection from the Remote Engine server to Redshift is successful or not.
If the problem persists, contact AWS Redshift support to check if there's an issue on their end.
