Does headless mode use multiple cores in v 3.0?


#1

I’ve just installed cellprofiler v3.0.0 on our cluster, noticed that in headless mode the displayed wall clock time is less than the cpu time.

I’ve only allocated 1 core per job on the scheduler, so this behavior strikes me as a bit odd, though I’ve not actually timed it.

Here’s an example of the cellprofiler command I’m running:

cellprofiler -r -c -p /path/to/pipeline.cppipe --data-file=/path/to/loaddata.csv -o /path/to/output_location

And what I see from stdout:

Tue Dec  5 13:16:32 2017: Image # 2, module LoadData # 1: CPU_time = 4.04 secs, Wall_time = 1.60 secs
Tue Dec  5 13:16:34 2017: Image # 2, module IdentifyPrimaryObjects # 2: CPU_time = 25.59 secs, Wall_time = 25.30 secs
Tue Dec  5 13:16:59 2017: Image # 2, module MeasureObjectIntensity # 3: CPU_time = 0.61 secs, Wall_time = 0.61 secs
Tue Dec  5 13:17:00 2017: Image # 2, module MeasureObjectSizeShape # 4: CPU_time = 15.44 secs, Wall_time = 15.41 secs
Tue Dec  5 13:17:15 2017: Image # 2, module MeasureGranularity # 5: CPU_time = 11.21 secs, Wall_time = 11.19 secs
Tue Dec  5 13:17:26 2017: Image # 2, module MeasureTexture # 6: CPU_time = 53.92 secs, Wall_time = 11.96 secs
Tue Dec  5 13:17:38 2017: Image # 2, module ExportToSpreadsheet # 7: CPU_time = 0.00 secs, Wall_time = 0.00 secs

#2

I’m tagging @mcquin since she’s been researching speed issues in the various versions of CellProfiler and might have some insight (or, maybe this points out an issue with how time is recorded, which could be problematic for our recent analyses!)


#3

Hi @Swarchal,

Here’s my understanding of “wall time” versus CPU time, and why might cause the “wall time” to be less than the CPU time:

“Wall time” is defined as the amount of time a process takes to run, as we observe it. Essentially, it’s the result you’d get with a stopwatch. The CPU time is the amount of time a process spends running on the CPU. If you have multiple CPUs (logical or physical), then CPU time is the total time the process spent on all of the CPUs (logical or physical).

Now, since I don’t know the details of your cluster, I don’t know if your CPUs support multiprocessing and I don’t know how CellProfiler was installed. But, given these reported times, I might guess that you have a multi-core processor and the version of numpy you have is optimized with OpenBLAS or something that will help numpy take advantage of that multi-core processor.

I hope this information helps you,
Claire


#4

Hi Claire,

Thanks for the response.

From reading that I think it’s to do with our cluster setup rather than anything cellprofiler related. I’ve since found some other python scripts that seems to ignore resource limits and use all available CPU cores on the node.

Thanks for the help,
Scott.