Multithreading - Cellprofiler 2.2 & 3.0

cores-cpu

#1

Hi All,

I started using Cellprofiler after small break. Sometime ago cellprofiler was able to use all cores in a processor and run multiple workers in parallel. I saw that this function is still maintained in 2.2 and 3.0 version of the program.

However, now I tested 2.2 and 3.0 versions (in GUI versions) and the program “says” that it runs multiple workers, but based on a CPU usage and Java window shows that only one set of images is being analysed. Which is quite disappointing since this slows down analysis very significantly. I tested this on three different PC in the lab and results are exactly the same. I am sure that not only I have problem, unless all PC here are configurated the same way.

My images are not very big, its only 9MB each, one set consist of six images like this. Thus I am confident that PC has more than enough of resources to handle more than one worker. I am using a workstation with 16 cores and 32GB of RAM memory. I split all the images into 10 groups and ran 10 individual Cellprofiler programs, which PC handled perfectly fine, however I feel that this is not a good solution.

I tried to change preferences in how many workers to run, from 1 to 16, but results are exactly the same all the time - only one worker running.

Therefore I am wondering is there anything I can do in order for Cellprofiler to use multiple cores, since this seems possible? Do I need to adjust any setting in Windows (I use Windows 10) or Cellprofiler?

Thank you in advance for the help!

Mindaugas


#2

I’m pretty sure that it works in 2.2.
Perhaps you have something specific in pipeline which blocks multiprocessing?
Try a simple pipeline with just one identify primary objects and check if it works.


#3

Hi Mindaugas,

I’m using 3.0 on my PC and don’t have that problem.
Just one idea - When you start the analysis, CellProfiler always runs through the whole pipeline once with a single worker before running with the maximum. Have you allowed it to go through the whole pipeline a couple of times?
Hope this helps,
Francesca


#4

Thank you for your responses and suggestions.

I tried running a simplified pipeline and there are no changes, it still runs one worker. I attached an image demonstrating that. So fafafft and nichollsfj, you observe that all (whatever it is) workers are being analysed at the same time? Since in my case its one after another, which is quite slow and doesn’t use full processor resources.

Let me know if someone has any suggestions!

Thank you for your help!


#5

Hi Mindaugas,
Screen shot attached for comparison (I only run 3 workers as that’s what seems to work best on my desktop PC), so I’m afraid I’m not sure why yours isn’t doing the same. If you want to drop me an email (I think you have it - Francesca in the Lovestone lab!), I’m happy to compare settings etc to see if we can figure it out.