Memory and Batch Processing


#1

I have been running CellProfiler to analyze images, but I keep getting an out-of-memory error when I try to analyze more than 70 images. I looked at the suggestions that have already been posted, but all of them either did not apply or did not help. Is there anything else I could do?

I am also trying to do batch processing for my CellProfiler pipeline. I looked at the example code for an LSF queing system, but we have a Sungrid queing system. Could you explain exactly what the code does so that I can modify it to work for our queing system?

Thank you very much!


#2

Hi,

Re: Out of memory error - There are three things that come to mind that could contribute to this error:

  • A large number of input images per cycle (e.g, multi-channel data)

  • Large images being processed in each cycle (e.g. very high-resolution)

  • A large number of objects being identified/measured in each cycle

The combination of any/all of these could be a culprit. Do any of them apply?

Re: queuing on Sungrid system - We don’t have much experience with those here, unfortunately. However, I should mention that the help for batch processing is fairly old (several years) and we have streamlined the process since then. You might be served by downloading the developer’s version of CP and taking a look at the BatchRunner script, which uses Python for the submission process.

Also, have you taken a look at the CPCluster portion of the forum? A user from this thread is also trying to get jobs submitted on a Sungrid system as well.

Hope this helps,
-Mark