Batch processing on a Single machine


#1

Hi Cell Profiler Users

I have a local machine where I have set up Cell Profiler(1.0.3755).
I want to use another(remote) machine, which has MATLAB to do batch processing.

My question is - Which Cell Profiler version do i copy onto the second machine - the same version used to build the scripts or can I use the MATLAB Developer version?

Also, I’d appreciate any suggestions on how to set up Batch processing on a single machine.

Thanks
Aravind


#2

Hi

Actually I ran the batch processing on the second machine (tried both the Developer’s and the ordinary version for win 32 xp)
When I run batch processing on the second machine,I get the following warning :-

Warning: Could not find appropriate function on path loading function handle >UNKNOWN Function

In NewBatch_2_to_16 at 6

and then the following error:-

Batch Error: LoadSingleImage Error using ==> feval
Undefined function or method ‘LoadSingleImage’ for input arguments of type ‘struct’.
??? Error using ==> feval

I checked out the handles.Function handle and it has the value @UNKNOWN. Could this be the problem? How could I overcome this?

Thanks
Aravind


#3

Hi Aravind,

First, have you followed all the directions in Help -> General Help -> Batch Processing? This walks you through the steps to get Batch Processing going, though usually on computing cluster (>1 computers). I haven’t tried batches with a single computer, but it should work.

The “cluster” computer should be running CPCluster, not the standard CellProfiler (see Downloads page). And the version # ought to match the version # of CellProfiler that created the batches. It may work with different version #s, but Module settings changes may cause CPCluster to error.

What is the architecture of your Batch machine (linux or ?). If linux, you can use the binary version without a Matlab license, but if not, then you should use the Developer’s version of CPCluster (remember to match the version #s!).

And make sure that the the CPCluster code has the same file structure as the local machine (see Step #2 in Help: BatchProcessing).

Hope this helps. Let us know!
Dave


#4

Hi David,

Thanks for your reply. I read the tutorial but my situation is a little odd and I am trying to do something that isn’t explained there.
Let me try to briefly explain what I am trying

My Basic Idea - If I could create the batch script files, I would be able to run the scripts on a target windows machine which has the Cell Profiler Modules and Matlab installed. To be clear, if the script created from Cell Profiler (non-developer version) on a local machine, was batchone.m - I wanted to execute batchone.m in the Matlab command Window on the target machine. Is this feasible?

The reason why I am trying this:-

a)I cannot run the batch processing scripts on my local machine - its windows and I do not have MATLAB on the local machine.(rules out CP Cluster developer version).
b) I have access to a target windows machine(a server) which has matlab. I don’t have admin rights on the target windows machine(rules out installing anything on it-but copying is possible).
c)So I’ve copied the cell profiler module files from my local machine to target machine. In short, I’m using Cell profiler(non-developer winxp32) on local to create script files and trying to run the script files on the target windows machine which has Matlab.

Regarding the Versions

The download page provides link only to one version. I faced some trouble installing the latest cell profiler version(for windows xp32 bit) - The MCR installer quits without installing and gives the following error message:-
‘The wizard was interrupted before Matlab Component Runtime 7.6 could be installed’

So I wasn’t able to install the latest version and I am using the version which I got from a colleague(v1.0.3775). I am not sure how, I can get the corresponding CP cluster Developer version.

I know my scenario is odd and probably confusing… but I hope you get the picture… I’d appreciate any pointers on setting up batch processing for my case.

Thanks again
Aravind


#5

Hi Aravind,

I think I am following you - please bear with me if I am not :smile:.

It sounds to me as though you should just dispense with the cluster version of CellProfiler and “Batch processing” altogether. You ought to be able to simply:

On the Local machine, with the Windows Binary version (since no Matlab):
(1) Create your pipeline on the local machine
(2) Save the pipeline (File->Save Pipeline)
(3) Manually move the .mat file to the Windows server

On the server, with the Windows Binary,
(4) Load .mat
(5) Run it.

No need for CreateBatchFiles Module at all, especially since there is only one server computer. And note that you would not need CPCluster - this is only for computing clusters that read in output files broken up into separate batch output .mat files by a CreateBatchFiles Module.

Perhaps the confusion is in the term “batch processing”? The standalone CellProfiler can load in multiple files and process each one through a pipeline, and only if you have an inordinate amount of images would you need to break up your overall job into separate jobs and send them off to a cluster of machines (our usage of “batch”).

One thing I don’t understand is your usage of “batchone.m” - do you mean batchone.mat"? The output files are .mat.

As for the MCR installer problem, are you logged in as an administrator? I just installed MCR with my laptop and it worked fine (after a reboot, that is). If this isn’t it, then I’ll have to ask around here. And you should not be using the CPCluster version as your primary CellProfiler program.

Let me know if this isn’t clear, or if it still doesn’t apply to your situation.
Cheers,
Dave


#6

Hi David,
Thanks again for your reply.
Your idea of just saving the pipeline in a .MAT and running it on the server using windows binary is a good one - but as I pointed out I have severe restrictions on the server - (and I forgot to mention)I cannot run the Cellprofiler or any other exe…

Regarding your question on batchone.m - I believe you can use Cell Profiler’s batch processing module to produce
a).MAT Files - if you plan to use the compiled version of CPCluster
b).M (Matlab Script) files - if you plan to use the MATLAB version of CPCluster

This was the first configurable option in the CreateBatchFiles Module in CellProfiler v1.0.3755. I will not be able to use the compiled version for two reasons(My server isnt linux and further, I cant run executables)

The only one thing I can run on the server is - a Matlab Script within Matlab Command Window. So my idea is to create the script files in local machine(where i have admin access and can install and run Cell profiler) and run them on the windows server within Matlab. I hope I am clearer this time :smile:

And strange enough, I am not able to install the MCR runtime. Let my try it on another computer and get back to you.

Thanks again!
Aravind


#7

Let me try to shed some light on what I’m trying. Actually what I am attempting to do is something like Batches of Batch Processing…

I am going to have different folders - each folder corresponding to a batch and I want to try to run a sequence of pipelines(each corresponding to one folder) one by one.

In an ideal case, I would create a set of Matlab Script files from CellProfiler binary(again one for each folder) and running the ‘Batches of batch’ would be the same as running multiple matlab scripts.

If I were to succeed in creating Multiple matlab scripts, I can set up a master script in Matlab easily which calls these multiple scripts one by one and happily watch the fun…

Then again, its proving to be tougher than I anticipated…

As I write, I am thinking whether it would be far easier to put all the files from different folders into one folder and set up a single batch… There was a logical need for separating the images and that is why they were present in different folders. But if it is going to be very hard for me to set my ‘batches of batch thing’ running, I might as well try overcoming the need to separate the images into folders.

Thanks again for your time
Aravind


#8

Hi,
I haven’t fully digested all your notes, but let me throw out one quick comment to make sure there’s not a simple misunderstanding… LoadImages has the option of processing multiple subfolders of images. So it is possible to keep your images in their proper subfolders and run the entire thing as one batch. David (and the rest of us) will spend some more time thinking about your questions and trying to figure out what they mean and post again, but let us know if that comment clears anything up!

I’m guessing ‘no’ and we are really going to have to think hard about this one.

:smile:

Anne


#9

Hi Anne,

Thanks for your reply. I didn’t know we could do that. If Load Images can read from multiple folders and write into multiple folders, it would be a good start for me - I would do all the work on my local machine…

Is this feature a new feature or has it been available in Load Images of older versions of Cell Profiler as well? Could you please tell me how to do use Load Images for reading from many sub-folders and write output files to corresponding sub-folders?

In the bigger picture though, I would like to be able to set up multiple pipelines - one for each folder and run them one by one on a local machine.

Once again, thanks for the insight.
Aravind


#10

Hi,
Well, that’s good news that this info helps you - indeed we have had this ‘subfolder’ feature ever since public release, I believe. The only problem with what you mention is that the output will be written all to a single output file (in the Default output folder). Another limitation (I believe) is that any processed images you save with a SaveImages module will be saved to a single output folder (you can change which folder in the module itself so it doesn’t have to be the main Default output folder, but subfolders are not an option as far as I recall. You might want to do a quick test because it’s possible we added subfolder-saving abilities recently, if it’s important to you.)

As to your your bigger-picture questions, a similar warning is described in these other topics:
[Undefined variable “cluster” in CPCluster, batch m)
[Creating pipeline on computer with different file system)
Sounds like the ‘undefined function’ warning is actually not the real problem, it might just be a little bug in the warning message itself. So that makes me think we should go after the ‘real’ error which has to do with feval. Doing a search on the Forum for feval led me to this, which seems to perhaps explain your issue and two possible solutions:
cellprofiler.org/forum/viewtopic … ight=feval
V1.0.4303 not listing modules and hangs)

Sounds like maybe you just need the latest download of CP? Or perhaps fix the Default Module directory in File > Set Preferences?

Let us know if any of those suggestions help. In the meantime, we will try to see whether we can get rid of the confusing ‘undefined function’ warning that is really just distracting.

Anne


#11

Hi Anne

Unfortunately, I’m not able to resolve the problem(my big picture).
I still keep getting the same error.

I downloaded and installed the latest cell profiler version,(a fix to my problem with Matlab component was described in the matlab site). Now, I have a problem running a pipeline on my local machine. The module MeasureObjectIntensity fails with the following error message - but the rest of the modules go through.

'There was a problem running the analysis module MeasureObjectIntensity which is number 16. Error using ==>var at 56,
First argument must be single or double…

Any idea what could be wrong here?

Thanks
Aravind


#12

Hi Aravind,
Are objects being identified in your image? There might be a bug where, if objects are not identified, CellProfiler does not know what to do if it is trying to measure the intensity of 0 objects.

~martha


#13

Hi Martha,

The objects are being detected. The same pipeline used to work with an older edition(1.03755) - I just had to make some changes to the pipeline when I changed versions. Though I get this error message, the other modules continue to run.

The CreateBatchFiles module also runs differently - I’m not able to create a .m script file. Though I specify the number of cycles to be far greater than the number of pictures in the batch, CellProfiler is running through the images one by one - instead of stopping after the first image and creating a script file and .mat file for the rest of the cycles.

Are there too many changes between the two versions that the pipeline created with one can’t be used with another?

Thanks
Aravind


#14

Hi Aravind,
I am surprised the the other measure modules are working. Is it possible that the image you selected for MeasureObjectIntensity is not in grayscale?
~martha


#15

Hi -
It’s probably also worth carefully checking every setting of the pipeline to make sure everything is precisely as it was in your old pipeline. It’s always a little painful loading an old pipeline and adapting it to a new version of CellProfiler and sometimes a few settings might be easily overlooked in the conversion.
Anne