Job hangs when running a COPY in Redshift as part of a larger job.
I've got a job that puts some data into Redshift and then puts some more data into Redshift in a later stage. For some reason, the second step hangs and I can't figure out why - I've checked that connections all have autocommit set to true and if I run the subjobs individually they all work fine. The only other thing I can think of is that there is a custom library I developed with functionality that is being loaded multiple times (once in each subjob). The issue is that the job gets to the point of the COPY (I've got a tjava ahead of it with system.out.println in it) and then freezes - I've left it for over an hour and nothing happens (it's insert a couple of rows and only takes a couple of seconds when I run the subjob separately). Is there any logging or anything I can turn on to track down the issue?
Re: Job hangs when running a COPY in Redshift as part of a larger job.
Sounds silly. Because of you can reproduce this problem I would use the Java Debugger to find out where exactly the job hangs. If you know the called method you should post again and we can try help you.