CP3.0.0 hangs in Analysis mode with custom plugin running subprocess.call()

subprocess

#1

Hi all, I’m currently writing a custom cellprofiler module that requires me to execute another program (in my case, ilastik). I’m doing this by calling subprocess.call(). In test mode, the code below works perfectly, but in analysis mode, CP3 hangs just before the Dummy module is executed, and the Dummy module is only executed after clicking on Stop Analysis (the rest of the workers are cancelled without executing their self.run()).

The cpproj consists of the usual four modules and a few images, plus the following module:

dummy.py

import subprocess
import cellprofiler.module
import cellprofiler.setting

FOO_PYTHON = ‘PATH_TO_PYTHON_EXE’
FOO_FILE = ‘PATH_TO_FOO_FILE’

class Dummy(cellprofiler.module.Module):
category = “Custom”
module_name = “Dummy”
variable_revision_number = 1

def create_settings(self):
    self.divider = cellprofiler.setting.Divider()

def settings(self):
    return [self.divider]

def run(self, workspace):
    subprocess.call([
        FOO_PYTHON,
        FOO_FILE
    ])

foo.py

print(‘from foo’)

The terminal displays:

Progress Counter({u’Unprocessed’: 2})
Starting workers on address tcp://127.0.0.1:57872
Progress Counter({u’Unprocessed’: 1, u’InProcess’: 1})
Worker 0: Starting job
Worker 0: Doing job: 1
Worker 0: Sun Jul 8 22:50:57 2018: Image # 1, module Images # 1: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Worker 0: Sun Jul 8 22:50:57 2018: Image # 1, module Metadata # 2: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Worker 0: Sun Jul 8 22:50:57 2018: Image # 1, module NamesAndTypes # 3: CPU_time = 2.92 secs, Wall_time = 2.68 secs
Worker 0: Sun Jul 8 22:50:59 2018: Image # 1, module Groups # 4: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Worker 0: from foo # logged just after Stop Analysis is clicked
Worker 0: Sun Jul 8 22:50:59 2018: Image # 1, module Dummy # 5: CPU_time = 0.00 secs, Wall_time = 10.88 secs
Worker 1: Cancelling worker
Worker 0: Aborting job after cancellation
Worker 0: Running module Images 1
Worker 0: Running module Metadata 2
Worker 0: Running module NamesAndTypes 3
Worker 0: Running module Groups 4
Worker 0: Running module Dummy 5
Worker 0: Cancelling worker
Progress Counter({u’Unprocessed’: 1, u’FinishedWaitingMeasurements’: 1})
Cancelled!

Is this something that’s possible in CP3? Any help would be greatly appreciated. Thanks!

-Sebastian


#2

@wsa, have you had a look at the *Predict* module? It also calls ilastik from within a CellProfiler pipeline. Perhaps the code in this file will guide you to a solution.


#3

It did! Thanks a ton!

Edit: I may have spoken too soon. Just found out about this issue: https://github.com/CellProfiler/CellProfiler-plugins/issues/19

Edit 2: I’ve managed to make it work, but only in headless mode and by adding the ‘–jvm-heap-size’ option back, to prevent the workers from running out of RAM.


#4

Hi @wsa, I’m glad you were able to find a work around. Thanks for sharing the results of your investigation.

I’ve referenced this thread in the github issue. If you end up finding a way to run this module from the gui, please consider submitting a pull request.