Installing CP on amazon ec2 ubuntu instance


#1

Hi, I’m trying to get a headless version of CP working on amazon EC2. I see in the forum I’m not the only one to have problems going down this path. I worked off of the instructions for ubuntu 14.04 wiki page: https://github.com/CellProfiler/CellProfiler/wiki/Source-Installation-(Ubuntu-14.04-LTS) Everything seems to have installed ok except when I go to run the mini example provided there are errors with java and Ilastik. Sorry if this sort of output belongs somewhere else.

Thanks for your help! I’ve used CP for years. Thanks for keeping it going! -John

Also, I tried the wiki for ubuntu 16.04 and again got CP installed, but could not run it headless.

ran this: cellprofiler -r -c -p ExampleHumanImages/ExampleHuman.cppipe --file-list=$PWD/filelist.txt -o .

Plugin directory doesn’t point to valid folder: /home/ubuntu/tmp/plugins
Version: 2016-05-03T18:31:00 ac0529e / 20160503183100
Failed to stop Ilastik
Pipeline saved with CellProfiler version 20140723174500
ilastik import: failed to import the ilastik. Please follow the instructions on
"http://www.ilastik.org" to install ilastik
Traceback (most recent call last):
File “/opt/CellProfiler/cellprofiler/modules/classifypixels.py”, line 71, in
from ilastik.core.dataMgr import DataMgr, DataItemImage
ImportError: No module named ilastik.core.dataMgr
ilastik import: failed to import the ilastik. Please follow the instructions on
"http://www.ilastik.org" to install ilastik
Traceback (most recent call last):
File “/opt/CellProfiler/cellprofiler/modules/ilastik_pixel_classification.py”, line 70, in
import ilastik_main
ImportError: No module named ilastik_main
Times reported are CPU times for each module, not wall-clock time
Fri Feb 10 12:44:59 2017: Image # 1, module Images # 1: 0.00 sec
Fri Feb 10 12:44:59 2017: Image # 1, module Metadata # 2: 0.00 sec
Warning: org.apache.xerces.parsers.SAXParser: Feature ‘http://javax.xml.XMLConstants/feature/secure-processing’ is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property ‘http://javax.xml.XMLConstants/property/accessExternalDTD’ is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property ‘http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit’ is not recognized.
Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:457)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:457)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:457)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Fri Feb 10 12:44:59 2017: Image # 1, module NamesAndTypes # 3: 1.40 sec
Fri Feb 10 12:45:00 2017: Image # 1, module Groups # 4: 0.00 sec
Error detected during run of module Crop
Traceback (most recent call last):
File “/opt/CellProfiler/cellprofiler/pipeline.py”, line 1819, in run_with_yield
self.run_module(module, workspace)
File “/opt/CellProfiler/cellprofiler/pipeline.py”, line 2067, in run_module
module.run(workspace)
File “/opt/CellProfiler/cellprofiler/modules/crop.py”, line 361, in run
internal_cropping)
File “/opt/CellProfiler/cellprofiler/cpimage.py”, line 466, in crop_image
return image[i_first:i_end,j_first:j_end].copy()
TypeError: only integer scalar arrays can be converted to a scalar index
Fri Feb 10 12:45:00 2017: Image # 1, module Crop # 5: 0.00 sec
Failed to stop Ilastik
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored


#2

Hi John,
Most of those errors are actually ‘expected’ ; the last one our software engineer thinks is due to a recent issue. Try updating your installation or switching to the stable branch (which should work).
We also have a CellProfiler docker that runs the stable branch that we use as part of our cloud cluster solution- see here for the main docker itself.


#3

Actually I was trying this because I went down the docker route and tried running a pipeline, but nothing happened and meaning no “Fleet” was launched. I was trying confirm my pipeline worked with the “file list” load option. I’m super interested in getting the docker approach working.

Anyway I did do the “stable” branch. These were my exact commands to install and run I attempted this morning:

history
2 sudo passwd
3 sudo timedatectl set-timezone America/Los_Angeles
4 date
5 sudo apt-get update
6 sudo apt-get install python-pip
7 sudo pip install awscli
8 aws configure
9 sudo apt-get install cython default-jdk git libmysqlclient-dev libhdf5-dev python-pip python-h5py python-matplotlib python-mysqldb python-scipy python-vigra python-lxml python-pandas python-scipy python-sklearn python-wxgtk2.8 python-zmq unzip
10 sudo -H pip install --upgrade cython
11 sudo -H pip install --upgrade scikit-image
12 sudo git clone https://github.com/CellProfiler/CellProfiler.git /opt/CellProfiler
13 cd /opt/CellProfiler
14 sudo git checkout stable
15 sudo pip install -e . --process-dependency-links

cd ~
mkdir ~/tmp && cd ~/tmp
wget http://d1zymp9ayga15t.cloudfront.net/content/Examplezips/ExampleHumanImages.zip && unzip ExampleHumanImages.zip
find $PWD/ExampleHumanImages -name “*.tif” > filelist.txt
cellprofiler -r -c -p ExampleHumanImages/ExampleHuman.cppipe --file-list=$PWD/filelist.txt -o .


#4

Can you try switching to master and seeing if that works?


#5

no, that did seem to help. did this:

cd /opt/CellProfiler
sudo git checkout master
sudo pip install -e . --process-dependency-links

then this:
ubuntu@ip-172-31-39-238:~/tmp$ cellprofiler -r -c -p ExampleHumanImages/ExampleHuman.cppipe --file-list=$PWD/filelist.txt -o .

but got this:

Your pipeline version is 20140723174500 but you are running CellProfiler version 300. Loading this pipeline may fail or have unpredictable results.
Times reported are CPU times for each module, not wall-clock time
Fri Feb 10 14:40:49 2017: Image # 1, module Images # 1: 0.00 sec
Fri Feb 10 14:40:49 2017: Image # 1, module Metadata # 2: 0.00 sec
Warning: org.apache.xerces.parsers.SAXParser: Feature ‘http://javax.xml.XMLConstants/feature/secure-processing’ is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property ‘http://javax.xml.XMLConstants/property/accessExternalDTD’ is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property ‘http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit’ is not recognized.
Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:457)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:457)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:457)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Fri Feb 10 14:40:49 2017: Image # 1, module NamesAndTypes # 3: 1.42 sec
Fri Feb 10 14:40:51 2017: Image # 1, module Groups # 4: 0.00 sec
/opt/CellProfiler/cellprofiler/image.py:410: VisibleDeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
return image[i_first:i_end, j_first:j_end].copy()
Fri Feb 10 14:40:51 2017: Image # 1, module Crop # 5: 0.01 sec
Fri Feb 10 14:40:51 2017: Image # 1, module Crop # 6: 0.01 sec
Fri Feb 10 14:40:51 2017: Image # 1, module Crop # 7: 0.01 sec
Error detected during run of module IdentifyPrimaryObjects
Traceback (most recent call last):
File “/opt/CellProfiler/cellprofiler/pipeline.py”, line 1753, in run_with_yield
self.run_module(module, workspace)
File “/opt/CellProfiler/cellprofiler/pipeline.py”, line 1997, in run_module
module.run(workspace)
File “/opt/CellProfiler/cellprofiler/modules/identifyprimaryobjects.py”, line 923, in run
binary_image = self.threshold_image(image_name, workspace, automatic=self.basic)
File “/opt/CellProfiler/cellprofiler/modules/identify.py”, line 748, in threshold_image
local_threshold, global_threshold = self.get_threshold(image, mask, workspace, automatic)
File “/opt/CellProfiler/cellprofiler/modules/identify.py”, line 849, in get_threshold
**kwparams
File “/usr/local/lib/python2.7/dist-packages/centrosome/threshold.py”, line 110, in get_threshold
threshold_method, image, mask, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/centrosome/threshold.py”, line 171, in get_global_threshold
raise NotImplementedError("%s algorithm not implemented"%(threshold_method))
NotImplementedError: None algorithm not implemented
Fri Feb 10 14:40:51 2017: Image # 1, module IdentifyPrimaryObjects # 8: 0.01 sec
Exception TypeError: TypeError("‘NoneType’ object is not callable",) in <bound method UmfpackContext.new_del of <scipy.sparse.linalg.dsolve.umfpack.umfpack.UmfpackContext object at 0x7fafe52ff9d0>> ignored

No apparent output:

ubuntu@ip-172-31-39-238:~/tmp$ ls -la
total 364
drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 10 12:43 .
drwxr-xr-x 7 ubuntu ubuntu 4096 Feb 10 12:42 …
drwxrwxr-x 2 ubuntu ubuntu 4096 Jul 25 2014 ExampleHumanImages
-rw-rw-r-- 1 ubuntu ubuntu 352743 Feb 4 2016 ExampleHumanImages.zip
-rw-rw-r-- 1 ubuntu ubuntu 213 Feb 10 12:43 filelist.txt
ubuntu@ip-172-31-39-238:~/tmp$ cd ExampleHumanImages/
ubuntu@ip-172-31-39-238:~/tmp/ExampleHumanImages$ ls -la
total 784
drwxrwxr-x 2 ubuntu ubuntu 4096 Jul 25 2014 .
drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 10 12:43 …
-rw-rw-r-- 1 ubuntu ubuntu 326 Jul 20 2010 AboutTheseImages.rtf
-rwxrwxr-x 1 ubuntu ubuntu 233336 Jul 20 2010 AS_09125_050116030001_D03f00d0.tif
-rwxrwxr-x 1 ubuntu ubuntu 240924 Jul 20 2010 AS_09125_050116030001_D03f00d1.tif
-rw-rw-r-- 1 ubuntu ubuntu 293652 Jul 20 2010 AS_09125_050116030001_D03f00d2.tif
-rw-rw-r-- 1 ubuntu ubuntu 17033 Jul 25 2014 ExampleHuman.cppipe


#6

sorry did not help


#7

There’s likely two sets of issues here

-Most of CP doesn’t work on numpy 1.12 - 1.12 broke compatability with a bunch of things apparently
-Master has a bunch of stuff being worked on right now as we reconfigure stuff getting ready for 3.0, which seems to have broken IdentifyPrimary for this pipeline (we removed some thresholding methods, so I’m guessing that’s why)

Can you try rolling your numpy installation to 1.11.2 and then see if stable works for you?