Max number of processors and RAM


#1

Hi,

I’m thinking about acquiring a Windows-based server to boost my image processing capabilities (I’m currently using CellProfiler on a 2012 MacPro and, since I’m also using Ilastik, I’m obliged to run Windows with Parallels…). Before deciding on what kind of machine to buy, my IT department is asking me:

  1. How many processors can CellProfiler handle?
  2. How much RAM can CellProfiler handle?
  3. Can I run CellProfiler (and Analyst) on Windows Server?

Thank you very much in advance!

Matteo


#2

I don’t know that there’s a strict upper limit of processors and/or RAM that CP can handle- it’ll spawn as many workers as you have CPUs. You may find many of the entries in the latter half of this thread helpful.

I searched the forum and found previous posts of people using CP on Windows Server, but nothing super-recent and none at all about Analyst in my brief search; as long as you can install 64 bit Java and JDK, I suspect they’ll work just fine, but I can’t say for certain.


#3

I eventually acquired and installed a workstation with 44 processors (88 with hyper threading) and 128GB of RAM. CP works great (with 44 workers) but slows down if I try to give it 80 workers. I THINK the main issue comes from the available RAM which seems to be limited at 64GB in the CP preference window. Is there a way to bypass this?

Thanks


#4

Hi,
It seems like the only way to do that is to install from source and manually change the preference in the source code. We’ll look at changing that in the 3.0 release. Sorry for the difficulty, let us know if that doesn’t work!


#5

uh…looks like something I’m NOT able to do…but I can ask the IT guys!!
In the meantime, would there be a point in giving CP 80 workers if I am limited to 64GB of ram of I would better keep the max number of workers at 64 so that each gets 1GB ram?


#6

In the meantime, would there be a point in giving CP 80 workers if I am limited to 64GB of ram of I would better keep the max number of workers at 64 so that each gets 1GB ram?

I honestly don’t know, I think you’d have to determine that empirically. Sorry, not too helpful today!


#7

alright…thanks anyway!


#8

Hello again,

I was playing around with the settings and something just occurred to me…but I’d like you to confirm this!
In the GUI preference pane, the setting about the amount of memory to dedicate to Java has nothing to do with the amount of memory we want to give our processors during analysis! Actually, INCREASING that number equals to SAVE ram for processes other than image analysis itself …so my interest is actually to keep that number as low as possible…is that right??


#9

Well, it kind of depends! A lot (though certainly not all) of CP’s heavily memory-dependent processes run through Java (see below). I personally don’t have a good sense of if increasing or decreasing the amount in your JVM vs Python VM will help, though you could always try halving your JVM setting right now and see if that increases your processing speed.


#10

well if I set the value to default (512MB) then I can see that my workstation dedicates a lot of its ram to image analysis, which is good. If I INCREASE the value then my workstation uses a lot less memory…
I’m happy with the performance on one 96-well plate (2h analysis vs the 9h I had with my previous computer…) but I’d like to run a pipeline I had created with my previous computer which allows me to analyze 13 96-well plates at the same time and this is where it gets tricky because now instead of having 26h analysis time (2h/plate * 13 plates) I have 150h analysis time… I have tried giving java more ram in the CP GUI but it does not seem to help that much…


#11

me again.
I got in touch with Derek (who posted about a year ago on the use of CP on a workstation) and he pointed me in the right direction. However I still have longer analysis time if I submit more than 1 plate at the time.
Any idea?