We had a similar problem where CellProfiler was killed on the cluster due to excessive memory usage.
However in our case, it turned out that the cluster was configured to limit virtual memory, not real memory.
Therefore this was a pretty strange (and actually useless) cluster setting, because it was much too
If you do not know the difference: virtual memory is like real memory, but additionally includes memory
locations that can be shared between applications. Also, the operating system does not need to actually
allocate all virtual memory. So its easily possible to run 50 applications with 4GB virtual memory each
on a machine that just has 16BG of RAM. With real memory, this would not work.
So just to double-check: are you certain its real memory (sometimes reported as RSS)? If yes, then its
actually strange and a problem. If its virtual memory, then this is absolutely acceptable, but you may want
to change the cluster configuration.