We have greatly benefited from CellProfiler in the past and are about to start processing another large set of images on a cluster. One of the changes from the past is that the cluster's /tmp directory now has a size limit of 500MB. As a result, running a few jobs at a time works fine, but everything fails due to lack of space once we increase the number of jobs.
I added the -t flag to all the commands, but from the error messages, it appears that CellProfiler is disregarding my instructions to use a different folder (/scratch/pmt3sw/temp) where there is plenty of space, and is still attempting to use /tmp as the temporary directory. I have replicated this problem several times, and the only possibilities I can think of are that either I am using the -t flag incorrectly, or that the program is ignoring it for some reason. Would you be able to provide any suggestions?
Thanks in advance!
Cardiac Systems Biology Lab
University of Virginia
Example of command calling CellProfiler:
python /home/pmt3sw/CellProfiler/CellProfiler.py --do-not-fetch -t /scratch/pmt3sw/temp -b -c -r -p /scratch/pmt3sw/control/DV005/batchE.h5 -f 1153 -l 1440
Relevant portion of the error message--close to the end it tries to use /tmp.
Thu Jul 23 15:34:18 2015: Image # 1154, module NamesAndTypes # 3: 0.37 sec
Uncaught exception in CellProfiler.py
Traceback (most recent call last):
File "/home/pmt3sw/CellProfiler/CellProfiler.py", line 279, in main
File "/home/pmt3sw/CellProfiler/CellProfiler.py", line 982, in run_pipeline_headless
File "/nv/blue/pmt3sw/CellProfiler/cellprofiler/measurements.py", line 359, in close
File "/share/apps/anaconda/2.2/lib/python2.7/site-packages/h5py/_hl/files.py", line 263, in flush
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py-2.4.0/h5py/_objects.c:2405)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py-2.4.0/h5py/_objects.c:2362)
File "h5py/h5f.pyx", line 109, in h5py.h5f.flush (-------src-dir-------/h5py-2.4.0/h5py/h5f.c:2044)
RuntimeError: Unable to flush file's cached information (File write failed: time = thu jul 23 15:34:19 2015
, filename = '/tmp/cellprofilerimagecachefb4pgg.h5', file descriptor = 231, errno = 28, error message = 'no space left on device', buf = 0x4b68198, total write size = 4096, bytes this sub-write = 4096, bytes actually written = 18446744073709551615, offset = 32097336)
Failed to stop Ilastik