Creating pipeline on computer with different file system


#1

For two reasons I’m trying to create the pipeline with the CreateBatchFiles module on my desktop linux computer and run batchrun.sh on a cluster that has a different file system from my desktop computer (of course, after copying the files created by clicking “Analyze images” to the cluster).

My home directory on the desktop is /home/bwittner and on the cluster it’s /shr/home/bwittner.

So, in the part of the CreateBatchFiles module where it asks for correspondence between the local machine and the cluster, I entered “/home” for the local machine part and “/shr/home” for the cluster machine part.

Nevertheless, when I ran on the cluster, I got the following error message:
Warning: Could not find appropriate function on path loading function handle /home/bwittner/cellp/CellProfiler_mcr/nfs/sabatini6_ata/2006_09_14_CPRelease4303/LINUX_CellProfiler/CellProfiler.m>LoadPipeline_Callback

Notice that it’s looking in /home/bwittner instead of /shr/home/bwittner, despite the fact that I tried to tell it how to translate from local to cluster.

Any help would be greatly appreciated.

-Ben

Ben Wittner
wittner.ben@mgh.harvard.edu
617-643-3166


Batch processing on a Single machine
#2

Hi Ben,

The warning you are getting is of no concern. In the handles structure, we save a handle to a function in CellProfiler.m, but this is not needed when running on the cluster. When MATLAB loads the Batch_data.mat, it see’s this incorrect handle and complains, but it does not stop processing.

Do you have CPCluster setup correctly on your cluster?

Mike


#3

The warning you are getting is of no concern

Thanks Mike.

“Do you have CPCluster setup correctly on your cluster?”

I don’t know. It seems to run, but I got no output for one of them. I thought it was becuase of the error I mentioned earlier, but now I see not. Probably it was because of the following error message:

In CPCluster at 6
Analyzing set 3
Elapsed time is 0.000543 seconds.
Batch Error: ExportToExcel Invalid handle object.
??? Invalid handle object.

Do you have any idea what might be going wrong there?

Thanks again.

-Ben


#4

Hi Ben,

I have never used ExportToExcel in a batch analysis, since the data set is usually too large to look at by eye and we export to a database instead. However, looking at the code, it looks like this might be an issue with the waitbar that is created while exporting measurements. Most likely, when run on the cluster, figures cannot be made and therefore the code crashes when it tries to update this waitbar. My suggestion is to remove this module from the analysis and try again. You can always add ExportToDatabase or manually export the excel files. I will be certain to fix this issue in the next release of CellProfiler.

Regards,
Mike


#5

Mike,

I have never used ExportToExcel in a batch analysis, since the data set is usually too large to look at by eye and we export to a database instead.

Our plan had been not to look at these by eye; rather we had planned to read them into a program to analyse.

I will be certain to fix this issue in the next release of CellProfiler.

Thanks.

You can always add ExportToDatabase or manually export the excel files.

Am I correct in assuming that manually exporting the files would mean manipulating the GUI once per well? If so, that would be too painful and we would probably try to figure out how to use the database.

Thanks again.
-Ben


#6

Hi Ben,

You are correct that you would have to use the GUI once per well. However, from the images I have looked at, it looks like you have each well in a different folder. Is there a reason for this? Could you put all the well’s in to one folder, analyze everything, then upload to a database? This is our method, and then we link each image to its condition (gene kd, chemical, etc.).

Mike


#7

it looks like you have each well in a different folder.
Is there a reason for this?

That’s how the sofware that runs our microscope, BD’s Attovision, outputs the images.

Could you put all the well’s in to one folder

We could if we have to, but Attovision and other BD software expects the images to be as Attovision created them, so we would like to avoid it if we can.

Could you put all the well’s in to one folder, analyze everything,
then upload to a database?

We also don’t understand how to analyze everything and then upload to a database for two reasons:

  1. We don’t think we know how to analyze everything and then do something and
  2. Those of us currently working on this are not familiar with databases, so we would have to enlist another person in the effort, set up a DBM and data base, etc.
    Our current idea is to use ExportToExcel on the local machine to get the column headers, which seem to match the SQL_1_1_object.CSV files that we can create on the cluster.

Thanks again. -Ben