Unable to change temporary directory on cluster


#1

Hello,

We have greatly benefited from CellProfiler in the past and are about to start processing another large set of images on a cluster. One of the changes from the past is that the cluster’s /tmp directory now has a size limit of 500MB. As a result, running a few jobs at a time works fine, but everything fails due to lack of space once we increase the number of jobs.

I added the -t flag to all the commands, but from the error messages, it appears that CellProfiler is disregarding my instructions to use a different folder (/scratch/pmt3sw/temp) where there is plenty of space, and is still attempting to use /tmp as the temporary directory. I have replicated this problem several times, and the only possibilities I can think of are that either I am using the -t flag incorrectly, or that the program is ignoring it for some reason. Would you be able to provide any suggestions?

Thanks in advance!

Philip Tan
Cardiac Systems Biology Lab
University of Virginia

Example of command calling CellProfiler:

Relevant portion of the error message–close to the end it tries to use /tmp.

Thu Jul 23 15:34:18 2015: Image # 1154, module NamesAndTypes # 3: 0.37 sec Uncaught exception in CellProfiler.py Traceback (most recent call last): File "/home/pmt3sw/CellProfiler/CellProfiler.py", line 279, in main run_pipeline_headless(options, args) File "/home/pmt3sw/CellProfiler/CellProfiler.py", line 982, in run_pipeline_headless measurements.close() File "/nv/blue/pmt3sw/CellProfiler/cellprofiler/measurements.py", line 359, in close self.__image_cache_file.flush() File "/share/apps/anaconda/2.2/lib/python2.7/site-packages/h5py/_hl/files.py", line 263, in flush h5f.flush(self.fid) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py-2.4.0/h5py/_objects.c:2405) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py-2.4.0/h5py/_objects.c:2362) File "h5py/h5f.pyx", line 109, in h5py.h5f.flush (-------src-dir-------/h5py-2.4.0/h5py/h5f.c:2044) RuntimeError: Unable to flush file's cached information (File write failed: time = thu jul 23 15:34:19 2015 , filename = '/tmp/cellprofilerimagecachefb4pgg.h5', file descriptor = 231, errno = 28, error message = 'no space left on device', buf = 0x4b68198, total write size = 4096, bytes this sub-write = 4096, bytes actually written = 18446744073709551615, offset = 32097336) Failed to stop Ilastik


Specifying a temporary directory via command line
#2

Hmm, we’ll take a look, but there are a couple issues we fixed in the past year or so that may be relevant. What version of CP on the cluster are you running?

-David


#3

I downloaded it from GitHub on 16 July 2015 using git clone https://github.com/CellProfiler/CellProfiler.git


#4

Thanks - I filed a bug here: github.com/CellProfiler/CellPro … ssues/1411 . As far as I can tell, it’s likely a bug, and we’ll report back here as soon as we can (or you can monitor the Github issue to be safe).

David


#5

Thanks! We appreciate the support.


#6

Hi Phillip,
Thanks for catching that bug. It’s good timing because it would have gone out in the release. You can pick up the changes from Github if you’re running from source or on the trunk builds page (cellprofiler.org/cgi_bin/trunk_build.cgi) in a couple hours once the build has run.

–Lee


#7

Thank you for providing such quick support! The new version is running perfectly. We really appreciate your help.

Philip


#8

Great, thanks for reporting back, Philip.
David