We can't get it right: 'High-throughput' headless pipeline on a cluster environment


#1

Dear all,

We aim to use CellProfiler in headless mode on a cluster for the automated analysis of large amounts of histology data. We have about twenty thousand slide images (TIFs of several GBs’) of several different stainings. These are masked and converted into 2000x2000 pixel tiles by a toolkit specifically designed for that purpose. The tiles are fed into CellProfiler pipelines for each respective staining.

…or that was the idea.

We had CellProfiler 2.x.x installed on our CentOS6 based High-Performance Cluster and some test runs (two years ago) were succesful. However, the cluster has recently been updated to CentOS7 and we needed to reinstall CP, which has been a nightmarish experience. From one side due to the fact that we do not have administrative permissions on our system, and the modular installation system (LMOD) employed by the HPC facility is unsuitable for installation of CP. From the other side, due to difficulties getting older releases of CP to work.

Because our pipelines were designed for a previous version of CellProfiler, we tried to install a previous version of CP. Git cloning a previous version commit and using make to build CP from source halts at a point where the script attempts to download dependency files (through an svn system) that are apparently no longer hosted / accessible on the website (connection time-outs).

We prefer to install CellProfiler in a contained environment so other users of the HPC facility can not meddle with our software, so ideally, it should not be dependent of the system wide Python installation. We therefore opted for the Anaconda method of installation. A lot of effort lead to an error-less installation, and correct display of --help and --version messages (although CP was still displaying rc3.0.0 as its version, whereas the actual commit dated back to version 2.1). Running a pipeline then, perhaps unsurprisingly, leads to a warning message about potentially unexpected behavior (as the pipeline was designed for a ‘previous’ version of CP). No error message is displayed, and CellProfiler quits without notice. Since there is no output, debug mode or verbose option, I have no idea what kind of error occurs.

From what I understand (and experienced), converted older pipelines do not work properly in headless mode out-of-the-box on a newer installation. CellProfiler (or at least the rc3.0.0) seemed to ignore the -i flag (in our tests), and search for the test directory specified in the pipeline/project file at the design stage.

We are quite desperate for a solution and no longer know how to continue searching for one. We would very much appreciate your advice in rethinking our approach,
considering:

  1. We need to run headless on a cluster, because of the overwhelming amount of data
  2. We cannot perform administrative tasks during installation, and installing all dependencies manually is a complete hell, so a no-go.
  3. We would preferably run 2.1 or maybe 2.2 to prevent having to alter the pipelines.

The best thing for us (we think) would be if the SVN would be restored, so we can attempt to build from source using a previous commit. This way, it should work as it did previously.

If this is not possible or if there is a better alternative: Please advise us on the best approach to set up such a new pipeline (e.g. What is the best installation strategy for our situation, What are important considerations/differences when loading data in a pipeline when compared to the design stage on a desktop?).

Thank you in advance!


#2

Howdy, @tbezemer:

If your system has a copy of the Java Development Kit (JDK), administrator privileges shouldn’t be required and you should be able to easily install CellProfiler’s dependencies with pip. However, I recommend uninstalling Anaconda as it’s clearly monkeying with your paths.

Clone the CellProfiler repository:

git clone git@github.com:CellProfiler/CellProfiler.git

Install CellProfiler and its dependencies:

cd CellProfiler
pip install --upgrade --user pip
pip install --upgrade --user cython
pip install --upgrade --user joblib
pip install --upgrade --user numpy
pip install --upgrade --user scipy
pip install --upgrade --user --editable .

#3

I’d also add to Allen’s comments that you should use CP 2.2+; a lot changed between 2.1.1 and 2.2 and my understanding is that the build process simplified in 2.2 and after. It should take you only a very few minutes on a workstation with a 2.2 GUI installed to open a 2.1.1 pipeline and re-save it, and will save you far more time in the long run.


#4

Thank you both for the helpful replies!

So, when I would want to install CP 2.2 from GitHub, which commit should I use? I don’t see the 2.2.x versions anywhere on this page. The current release rc3.0.0 on GitHub is not stable enough for production environments, correct?


#5

There’s a branch called “stable” that I use for production-scale headless use- master is currently pretty unstable, yes, so I’d hesitate to recommend that at the moment.


#6

I have made some progress on installing the “stable” branch. Unfortunately, we are stuck with anaconda, because our default Python installation is system-wide and maintained by the administrators. The HPC facility is a multi-faculty facility and they do not install packages that are only used by specific groups. Pip in anaconda allows us to install packages only within the anaconda subdirectory, which is located in a folder that we have full control over.

When I try to run the pipeline now I get the following error:

vigra import: failed to import the vigra library. Please follow the instructions on
"http://hci.iwr.uni-heidelberg.de/vigra/" to install vigra
Traceback (most recent call last):
  File "/hpc/local/CentOS7/dhl_ec/software/anaconda/CellProfiler/cellprofiler/modules/classifypixels.py", line 50, in <module>
    import vigra
ImportError: No module named vigra
vigra import: failed to import the vigra library. Please follow the instructions on
"http://hci.iwr.uni-heidelberg.de/vigra/" to install vigra
Traceback (most recent call last):
  File "/hpc/local/CentOS7/dhl_ec/software/anaconda/CellProfiler/cellprofiler/modules/ilastik_pixel_classification.py", line 47, in <module>
    import vigra
ImportError: No module named vigra

The link in the error to the installation instructions does not work, and I know that vigra has something to do with the Ilastik module implemented in CP, so I wanted to ask you for advice before proceeding.


#7

Can you try to conda install vigra? It looks like it should be possible.


#8

I tried that this morning, the following (rather lengthy, my apologies) error is thrown:
I have also tried to update matplotlib as installing vigra led to downgrading of matplotlib, and I thought that might cause some of the problems below. It didn’t help or change the output.

Version: 2016-05-03T18:31:00 ac0529e / 20160503183100
Failed to stop Ilastik
Pipeline saved with CellProfiler version 20160308191757

Bad key "patch.force_edgecolor" on line 33 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "lines.dashed_pattern" on line 18 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "boxplot.meanprops.markeredgecolor" on line 369 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "date.autoformatter.second" on line 235 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "ytick.major.right" on line 271 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "date.autoformatter.month" on line 231 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "scatter.marker" on line 345 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "ytick.minor.right" on line 273 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "ytick.major.left" on line 270 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "ytick.left" on line 258 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "xtick.minor.bottom" on line 256 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "axes.autolimit_mode" on line 220 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "hatch.color" on line 37 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "boxplot.meanprops.marker" on line 367 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "date.autoformatter.minute" on line 234 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "xtick.major.top" on line 253 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "date.autoformatter.day" on line 232 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "ytick.right" on line 259 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "xtick.bottom" on line 242 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "date.autoformatter.microsecond" on line 236 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "hist.bins" on line 40 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "lines.scale_dashes" on line 21 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "axes.titlepad" on line 184 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "_internal.classic_mode" on line 526 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "date.autoformatter.hour" on line 233 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "boxplot.meanprops.markerfacecolor" on line 368 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "date.autoformatter.year" on line 230 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/__init__.py:1069: UserWarning: Bad val "auto" on line #360
	"boxplot.flierprops.markerfacecolor: auto
"
	in file "/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle"
	Key boxplot.flierprops.markerfacecolor: auto does not look like a color arg
  (val, error_details, msg))

Bad key "lines.dotted_pattern" on line 20 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "axes.formatter.offset_threshold" on line 207 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "xtick.minor.top" on line 255 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "ytick.minor.left" on line 272 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "lines.dashdot_pattern" on line 19 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "xtick.major.bottom" on line 254 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "boxplot.meanprops.markersize" on line 370 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "hatch.linewidth" on line 38 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution

Bad key "xtick.top" on line 241 in
/hpc/local/CentOS7/dhl_ec/software/anaconda/lib/python2.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution
ilastik import: failed to import the ilastik. Please follow the instructions on
    "http://www.ilastik.org" to install ilastik
Traceback (most recent call last):
  File "/hpc/local/CentOS7/dhl_ec/software/anaconda/CellProfiler/cellprofiler/modules/classifypixels.py", line 71, in <module>
    from ilastik.core.dataMgr import DataMgr, DataItemImage
ImportError: No module named ilastik.core.dataMgr
ilastik import: failed to import the ilastik. Please follow the instructions on
                          "http://www.ilastik.org" to install ilastik
Traceback (most recent call last):
  File "/hpc/local/CentOS7/dhl_ec/software/anaconda/CellProfiler/cellprofiler/modules/ilastik_pixel_classification.py", line 70, in <module>
    import ilastik_main
ImportError: No module named ilastik_main
Failed to stop Ilastik

#9

Sounds like your matplotlib installation may be out of date; can you post the output of pip freeze?
ETA Sorry, reading failure, I got bogged down in the stack trace; I still wonder if it’ll be possible to update matplotlib even after vigra has downgraded it, but it’d still be helpful to know from your freeze what version you’re at.


#10

I edited my reply to include that I updated the matplotlib installation.

cellh5==1.3.0
-e git+https://github.com/CellProfiler/CellProfiler.git@ac0529ec5b243cc8fce0c7de43cc2c29119e29c5#egg=CellProfiler
centrosome==1.0.7
cffi==1.9.1
conda==4.3.17
cryptography==1.7.1
cycler==0.10.0
decorator==4.0.11
enum34==1.1.6
functools32==3.2.3.post2
h5py==2.7.0
hmmlearn==0.2.0
idna==2.2
inflect==0.2.5
ipaddress==1.0.18
javabridge==1.0.14
libtiff==0.4.0
lxml==3.7.3
matplotlib==2.0.1
MySQL-python==1.2.5
networkx==1.11
numpy==1.12.1
olefile==0.44
pandas==0.17.1
Pillow==4.1.1
prokaryote==1.0.11
py==1.4.33
pyasn1==0.1.9
pycairo==1.10.0
pycosat==0.6.1
pycparser==2.17
pyOpenSSL==16.2.0
pyparsing==2.2.0
pytest==3.0.7
python-bioformats==1.1.0
python-dateutil==2.6.0
pytz==2017.2
PyWavelets==0.5.2
pyzmq==16.0.2
requests==2.12.4
scikit-image==0.13.0
scikit-learn==0.17.1
scipy==0.19.0
six==1.10.0
subprocess32==3.2.7
wxPython==3.0.0.0
wxPython-common==3.0.0.0

#11

Yeah, I was doubting it would help too. Usually downgrading happens for a reason…


#12

Sorry for spamming, but I also felt the output of conda list could be helpful:

# packages in environment at /hpc/local/CentOS7/dhl_ec/software/anaconda:
#
boost                     1.63.0              np111py27_6    conda-forge
boost-cpp                 1.63.0                        2    conda-forge
bzip2                     1.0.6                         1    conda-forge
cairo                     1.14.8                        0
cellh5                    1.3.0                     <pip>
centrosome                1.0.7                     <pip>
cffi                      1.9.1                    py27_0
conda                     4.3.17                   py27_0
conda-env                 2.6.0                         0
cryptography              1.7.1                    py27_0
cycler                    0.10.0                    <pip>
cycler                    0.10.0                   py27_0
dbus                      1.10.10                       0
decorator                 4.0.11                    <pip>
enum34                    1.1.6                    py27_0
expat                     2.1.0                         0
fftw                      3.3.6                         0    conda-forge
fontconfig                2.12.1                        3
freetype                  2.5.5                         2
functools32               3.2.3.2                  py27_0
functools32               3.2.3.post2               <pip>
glib                      2.50.2                        1
gst-plugins-base          1.8.0                         0
gstreamer                 1.8.0                         0
h5py                      2.7.0                     <pip>
hdf5                      1.8.17                       10    conda-forge
hmmlearn                  0.2.0                     <pip>
icu                       58.1                          1    conda-forge
idna                      2.2                      py27_0
inflect                   0.2.5                     <pip>
ipaddress                 1.0.18                   py27_0
javabridge                1.0.14                    <pip>
jpeg                      9b                            0
libffi                    3.2.1                         1
libgcc                    5.2.0                         0
libgfortran               3.0.0                         1
libiconv                  1.14                          0
libpng                    1.6.27                        0
libtiff                   0.4.0                     <pip>
libtiff                   4.0.6                         7    conda-forge
libxcb                    1.12                          1
libxml2                   2.9.4                         0
lxml                      3.7.3                     <pip>
matplotlib                2.0.1                     <pip>
matplotlib                1.5.1               np111py27_0
mkl                       2017.0.1                      0
MySQL-python              1.2.5                     <pip>
networkx                  1.11                      <pip>
numpy                     1.11.3                   py27_0
numpy                     1.12.1                    <pip>
olefile                   0.44                      <pip>
openssl                   1.0.2k                        0
pcre                      8.39                          1
Pillow                    4.1.1                     <pip>
pip                       9.0.1                    py27_1
pixman                    0.34.0                        0
prokaryote                1.0.11                    <pip>
py                        1.4.33                    <pip>
pyasn1                    0.1.9                    py27_0
pycairo                   1.10.0                   py27_0
pycosat                   0.6.1                    py27_1
pycparser                 2.17                     py27_0
pyopenssl                 16.2.0                   py27_0
pyparsing                 2.2.0                     <pip>
pyparsing                 2.1.4                    py27_0
pyqt                      4.11.4                   py27_2    conda-forge
pytest                    3.0.7                     <pip>
python                    2.7.13                        0
python-bioformats         1.1.0                     <pip>
python-dateutil           2.6.0                    py27_0
python-dateutil           2.6.0                     <pip>
pytz                      2017.2                   py27_0
pytz                      2017.2                    <pip>
PyWavelets                0.5.2                     <pip>
pyzmq                     16.0.2                    <pip>
qt                        4.8.7                         3
readline                  6.2                           2
requests                  2.12.4                   py27_0
ruamel_yaml               0.11.14                  py27_1
scikit-image              0.13.0                    <pip>
scipy                     0.19.0                    <pip>
setuptools                27.2.0                   py27_0
sip                       4.18                     py27_0
six                       1.10.0                   py27_0
sqlite                    3.13.0                        0
subprocess32              3.2.7                    py27_0
subprocess32              3.2.7                     <pip>
tk                        8.5.18                        0
vigra                     1.11.0             np111py27_11    conda-forge
wheel                     0.29.0                   py27_0
wxpython                  3.0.0.0                  py27_2
xz                        5.2.2                         0    conda-forge
yaml                      0.1.6                         0
zlib                      1.2.8                         3

#13

Ah, I have a suspicion it’s actually the opposite problem; I’m not sure if CP works with matplotlib 2.0 (I’m trying to get verification now). Can you roll your matplotlib back to 1.5 and see if that helps?


#14

Conda and pip both had a version of matplotlib installed (As detailed in the overviews above). I uninstalled both and reinstalled 1.5.1 using conda.

The error message has now condensed to this:

CellProfiler: Processing , in directory /home/dhl_ec/tbezemer/SLIDES/GLYCC/ae_tifs/.tiles, outputting in /home/dhl_ec/tbezemer/SLIDES/GLYCC/ae_tifs/output
Version: 2016-05-03T18:31:00 ac0529e / 20160503183100
Failed to stop Ilastik
Pipeline saved with CellProfiler version 20160308191757
ilastik import: failed to import the ilastik. Please follow the instructions on
    "http://www.ilastik.org" to install ilastik
Traceback (most recent call last):
  File "/hpc/local/CentOS7/dhl_ec/software/anaconda/CellProfiler/cellprofiler/modules/classifypixels.py", line 71, in <module>
    from ilastik.core.dataMgr import DataMgr, DataItemImage
ImportError: No module named ilastik.core.dataMgr
ilastik import: failed to import the ilastik. Please follow the instructions on
                          "http://www.ilastik.org" to install ilastik
Traceback (most recent call last):
  File "/hpc/local/CentOS7/dhl_ec/software/anaconda/CellProfiler/cellprofiler/modules/ilastik_pixel_classification.py", line 70, in <module>
    import ilastik_main
ImportError: No module named ilastik_main
Failed to stop Ilastik

So I think the matplotlib issue is now indeed resolved - Thanks!


#15

That error message actually comes up no matter what- it sounds like you’ve got it correctly installed now!


#16

That is good to know, thanks a lot! I have been at this stage numerous times before, where just the Ilastik error message was left. I didn’t know, so it’s reassuring to hear that it should be okay now.

However, CP quits right after that message and doesn’t return any output.
This is the command I use:

cellprofiler -p glycophorin_renewed.cppipe -i $CP_INPUT_DIR -o $CP_OUTPUT_DIR -c -r

Am I still missing something?


#17

You need to pass a file list or a CSV; as of 3.0 you can just point CP to an input folder and it will just work, but in 2.2 you need to explicitly tell it the files you want it to run.


#18

Thank you. Wow, that explains a lot! I am still struggling though. I tried setting --file-list to a file containing full paths to the files, as well as setting the -i flag to point to the image directory, and setting --file-list to a file containing all local paths to the images. Both yielded the same result as before (Ilastik error and quitting right after that).

Is there by chance any hidden development setting I can use to make CellProfiler a little more verbose about what’s happening under the hood, so I get an impression what’s wrong my parameter settings?


#19

Can you upload your pipeline, and also if possible a (sanitized) version of a small piece of your file list? My guess is that they somehow don’t match up correctly and/or that there’s an error in your pipeline somewhere.


#20

Hi Beth,

It could be that there is an abnormality in the pipeline file. They have been on the shelf for a long time since primary use.

File list is of the form:

AE1000.[PATIENT_ID].GLYCC.20150928.X0.Y0.tile.tissue.png
...
AE1000.[PATIENT_ID].GLYCC.20150928.X10.Y10.tile.tissue.png

or, for the full paths:

/home/dhl_ec/tbezemer/SLIDES/GLYCC/ae_tifs/AE1000.[PATIENT_ID].GLYCC.20150928/AE1000.[PATIENT_ID].GLYCC.20150928.tiles/AE1000.[PATIENT_ID].GLYCC.20150928.X0.Y0.tile.tissue.png
...
/home/dhl_ec/tbezemer/SLIDES/GLYCC/ae_tifs/AE1000.[PATIENT_ID].GLYCC.20150928/AE1000.[PATIENT_ID].GLYCC.20150928.tiles/AE1000.[PATIENT_ID].GLYCC.20150928.X10.Y10.tile.tissue.png

And the pipeline:

CellProfiler Pipeline: http://www.cellprofiler.org
Version:1
SVNRevision:11710

LoadImages:[module_num:1|svn_version:\'Unknown\'|variable_revision_number:11|show_window:False|notes:\x5B\'# Pipeline created for HPC (b.g.l.nelissen@umcutrecht.nl)\', \'# Goal\x3A Measure total tissue area and DAB area\', \'# Outlines\x3A Tissue blue, DAB red\', \'\'\x5D]
    File type to be loaded:individual images
    File selection method:Text-Exact match
    Number of images in each group?:3
    Type the text that the excluded images have in common:.counted
    Analyze all subfolders within the selected folder?:All
    Input image file location:Default Input Folder\x7CNone
    Check image sets for missing or duplicate files?:No
    Group images by metadata?:No
    Exclude certain files?:Yes
    Specify metadata fields to group by:
    Select subfolders to analyze:
    Image count:1
    Text that these images have in common (case-sensitive):tissue
    Position of this image in each group:1
    Extract metadata from where?:File name
    Regular expression that finds metadata in the file name:^(?P<STUDYNR>\x5B^.\x5D*)\\.(?P<STAIN>\x5B^.\x5D*).*\\.X(?P<X>\x5B0-9\x5D{1,4}).*\\.Y(?P<Y>\x5B0-9\x5D{1,4})
    Type the regular expression that finds metadata in the subfolder path:.*\x5B\\\\/\x5D(?P<Date>.*)\x5B\\\\/\x5D(?P<Run>.*)$
    Channel count:1
    Group the movie frames?:No
    Grouping method:Interleaved
    Number of channels per group:3
    Load the input as images or objects?:Images
    Name this loaded image:Original
    Name this loaded object:Nuclei
    Retain outlines of loaded objects?:No
    Name the outline image:NucleiOutlines
    Channel number:1
    Rescale intensities?:No

ColorToGray:[module_num:2|svn_version:\'Unknown\'|variable_revision_number:2|show_window:False|notes:\x5B\'Create Grayscale image for finding total tissue area\'\x5D]
    Select the input image:Original
    Conversion method:Combine
    Image type\x3A:RGB
    Name the output image:OrigGray
    Relative weight of the red channel:1
    Relative weight of the green channel:1
    Relative weight of the blue channel:1
    Convert red to gray?:Yes
    Name the output image:OrigRed
    Convert green to gray?:Yes
    Name the output image:OrigGreen
    Convert blue to gray?:Yes
    Name the output image:OrigBlue
    Channel count:1
    Channel number\x3A:Red\x3A 1
    Relative weight of the channel:1
    Image name\x3A:Channel1

Smooth:[module_num:3|svn_version:\'Unknown\'|variable_revision_number:1|show_window:False|notes:\x5B\'Smooth grayscale image to remove dust and small open holes\'\x5D]
    Select the input image:OrigGray
    Name the output image:OrigGraySmooth
    Select smoothing method:Gaussian Filter
    Calculate artifact diameter automatically?:No
    Typical artifact diameter, in  pixels:20
    Edge intensity difference:0.1

ImageMath:[module_num:4|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Invert grayscale image for futher analysis\'\x5D]
    Operation:Invert
    Raise the power of the result by:1
    Multiply the result by:1
    Add to result:0
    Set values less than 0 equal to 0?:Yes
    Set values greater than 1 equal to 1?:Yes
    Ignore the image masks?:No
    Name the output image:Inverted
    Image or measurement?:Image
    Select the first image:OrigGraySmooth
    Multiply the first image by:1
    Measurement:
    Image or measurement?:Image
    Select the second image:
    Multiply the second image by:1
    Measurement:

IdentifyPrimaryObjects:[module_num:5|svn_version:\'Unknown\'|variable_revision_number:8|show_window:False|notes:\x5B\'Find tissue using the grayscale image.\', \'No use of magick here; white is background, non-white is foreground.\', \'From now on, outlined foreground is called Tissue\', \'Diameter is choosen after some testing\'\x5D]
    Select the input image:Inverted
    Name the primary objects to be identified:Tissue
    Typical diameter of objects, in pixel units (Min,Max):40,99999999
    Discard objects outside the diameter range?:Yes
    Try to merge too small objects with nearby larger objects?:No
    Discard objects touching the border of the image?:No
    Select the thresholding method:Manual
    Threshold correction factor:0.9
    Lower and upper bounds on threshold:0.000000,1.000000
    Approximate fraction of image covered by objects?:0.01
    Method to distinguish clumped objects:None
    Method to draw dividing lines between clumped objects:Intensity
    Size of smoothing filter:10
    Suppress local maxima that are closer than this minimum allowed distance:7
    Speed up by using lower-resolution image to find local maxima?:Yes
    Name the outline image:Tissue
    Fill holes in identified objects?:No
    Automatically calculate size of smoothing filter?:Yes
    Automatically calculate minimum allowed distance between local maxima?:Yes
    Manual threshold:0.05
    Select binary image:None
    Retain outlines of the identified objects?:Yes
    Automatically calculate the threshold using the Otsu method?:Yes
    Enter Laplacian of Gaussian threshold:0.5
    Two-class or three-class thresholding?:Three classes
    Minimize the weighted variance or the entropy?:Weighted variance
    Assign pixels in the middle intensity class to the foreground or the background?:Foreground
    Automatically calculate the size of objects for the Laplacian of Gaussian filter?:Yes
    Enter LoG filter diameter:5
    Handling of objects if excessive number of objects identified:Continue
    Maximum number of objects:500
    Select the measurement to threshold with:None

MeasureImageAreaOccupied:[module_num:6|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Measure all Tissue areas\'\x5D]
    Hidden:1
    Measure the area occupied in a binary image, or in objects?:Objects
    Select objects to measure:Tissue
    Retain a binary image of the object regions?:No
    Name the output binary image:Stain
    Select a binary image to measure:Tissue

UnmixColors:[module_num:7|svn_version:\'Unknown\'|variable_revision_number:2|show_window:True|notes:\x5B\'Unmix the colors, and extract DAB\', \'Create one grayscale DAB channel\'\x5D]
    Stain count:1
    Color image:Original
    Image name:DAB
    Stain:DAB
    Red absorbance:0.363092
    Green absorbance:0.651435
    Blue absorbance:0.666181

MaskImage:[module_num:8|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Use selected Tissue object to create a mask\', "Only analyse what\'s in the \'Tissue\' area"\x5D]
    Select the input image:DAB
    Name the output image:DAB_masked
    Use objects or an image as a mask?:Objects
    Select object for mask:Tissue
    Select image for mask:Tissue
    Invert the mask?:No

IdentifyPrimaryObjects:[module_num:9|svn_version:\'Unknown\'|variable_revision_number:8|show_window:False|notes:\x5B"Find DAB positive area\'s", \'You need to test which thresholding method works best for you. Read the help menu for more info.\'\x5D]
    Select the input image:DAB_masked
    Name the primary objects to be identified:DAB_area
    Typical diameter of objects, in pixel units (Min,Max):8,4000000
    Discard objects outside the diameter range?:Yes
    Try to merge too small objects with nearby larger objects?:No
    Discard objects touching the border of the image?:No
    Select the thresholding method:RobustBackground Global
    Threshold correction factor:1
    Lower and upper bounds on threshold:0,1.0
    Approximate fraction of image covered by objects?:0.01
    Method to distinguish clumped objects:None
    Method to draw dividing lines between clumped objects:Intensity
    Size of smoothing filter:10
    Suppress local maxima that are closer than this minimum allowed distance:7
    Speed up by using lower-resolution image to find local maxima?:Yes
    Name the outline image:DAB_area
    Fill holes in identified objects?:No
    Automatically calculate size of smoothing filter?:Yes
    Automatically calculate minimum allowed distance between local maxima?:Yes
    Manual threshold:0.8
    Select binary image:None
    Retain outlines of the identified objects?:Yes
    Automatically calculate the threshold using the Otsu method?:Yes
    Enter Laplacian of Gaussian threshold:0.5
    Two-class or three-class thresholding?:Two classes
    Minimize the weighted variance or the entropy?:Weighted variance
    Assign pixels in the middle intensity class to the foreground or the background?:Background
    Automatically calculate the size of objects for the Laplacian of Gaussian filter?:Yes
    Enter LoG filter diameter:5
    Handling of objects if excessive number of objects identified:Continue
    Maximum number of objects:500
    Select the measurement to threshold with:None

MeasureImageAreaOccupied:[module_num:10|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Measure all DAB areas (within Tissue)\'\x5D]
    Hidden:1
    Measure the area occupied in a binary image, or in objects?:Objects
    Select objects to measure:DAB_area
    Retain a binary image of the object regions?:No
    Name the output binary image:Stain
    Select a binary image to measure:Tissue

OverlayOutlines:[module_num:11|svn_version:\'Unknown\'|variable_revision_number:2|show_window:False|notes:\x5B\'Create outlines for visualisation.\', \'The Tissue outline has a thick blue like\'\x5D]
    Display outlines on a blank image?:No
    Select image on which to display outlines:Original
    Name the output image:Orig_Overlay
    Select outline display mode:Color
    Select method to determine brightness of outlines:Max of image
    Width of outlines:4
    Select outlines to display:Tissue
    Select outline color:Blue

OverlayOutlines:[module_num:12|svn_version:\'Unknown\'|variable_revision_number:2|show_window:False|notes:\x5B\'Create outlines for visualisation.\', \'The DAB outline has a thin green line\'\x5D]
    Display outlines on a blank image?:No
    Select image on which to display outlines:Orig_Overlay
    Name the output image:Orig_Overlay
    Select outline display mode:Color
    Select method to determine brightness of outlines:Max of image
    Width of outlines:1
    Select outlines to display:DAB_area
    Select outline color:Red

SaveImages:[module_num:13|svn_version:\'Unknown\'|variable_revision_number:7|show_window:False|notes:\x5B\'Save original images with outlines so we can visually check what the pipeline did.\'\x5D]
    Select the type of image to save:Image
    Select the image to save:Orig_Overlay
    Select the objects to save:None
    Select the module display window to save:None
    Select method for constructing file names:From image filename
    Select image name for file prefix:Original
    Enter single file name:OrigBlue
    Do you want to add a suffix to the image file name?:Yes
    Text to append to the image name:.overlay
    Select file format to use:png
    Output file location:Default Output Folder\x7CNone
    Image bit depth:8
    Overwrite existing files without warning?:Yes
    Select how often to save:Every cycle
    Rescale the images? :No
    Save as grayscale or color image?:Grayscale
    Select colormap:gray
    Store file and path information to the saved image?:No
    Create subfolders in the output folder?:No