Extraction of metadata fails for a tiff stack

tiff
metadata

#1

I’m trying to extract metadata from this tiff stack. In Metadata module, I first click “Update metadata”, then “Update” and I obtain this error:

Traceback (most recent call last):
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/gui/cornerbuttonmixin.py”, line 113, in on_corner_left_mouse_up
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/gui/moduleview.py”, line 3931, in on_corner_button_clicked
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/modules/metadata.py”, line 916, in update_table
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/pipeline.py”, line 2725, in get_image_plane_details
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/pipeline.py”, line 2691, in __prepare_run_module
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/modules/metadata.py”, line 731, in prepare_run
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/javabridge/jutil.py”, line 885, in call
File “/Users/xxx/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/javabridge/jutil.py”, line 852, in fn
JavaException: java.lang.IllegalArgumentException: -2.77894E9 must not be null or non-negative.

My CP is 2.2.0 (rev 9969f42) on OSX 10.11.6.

The file loads correctly in Fiji.


Failure to load OME-TIFF sample data
#2

Here’s a probably related problem. This OME-TIFF stack created with BioFormats’ bfconvert loads to Fiji as T=25, C=2, Z=1. However, when I load it to CP, metadata extraction produces T=50, C=1, Z=1.

The message in command line is:

Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType


#3

Hmmm, I don’t get that error, it works fine for me- can you give us more information on your CP version, OS and version, etc?

If worst came to worst, you could always export your images as individual tifs rather than a single large one (it’d probably be easier to extract metadata that way), but lets not jump to that as a first solution.


#4

I tried it on 3 different installations:

  • 2.2.0 (rev 9969f42), OSX 10.11.6, Java 8 Update 111,
    java version "1.8.0_91"
    Java™ SE Runtime Environment (build 1.8.0_91-b14)
    Java HotSpot™ 64-Bit Server VM (build 25.91-b14, mixed mode)

  • 3.0.0rc1 (Nightly), OSX 10.11.6, java as above

  • 2.2.0 (rev ac0529e), Ubuntu 16.04.1 LTE,
    openjdk version "1.8.0_111"
    OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14)
    OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

The following terminal messages/errors are from the 3rd installation above (Ubuntu).


This stack, after clicking “Update metadata” in Metadata module:

Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType

and then after clicking “Update”:

Exception in thread “Thread-0” loci.common.services.ServiceException: java.lang.IllegalArgumentException: -2.77894E9 must not be null or non-negative.
at loci.formats.services.OMEXMLServiceImpl.createRoot(OMEXMLServiceImpl.java:384)
at loci.formats.services.OMEXMLServiceImpl.createOMEXMLRoot(OMEXMLServiceImpl.java:350)
at org.cellprofiler.imageset.ImageFile.setXMLDocument(ImageFile.java:166)
at org.cellprofiler.imageset.ImagePlaneMetadataExtractor.extract(ImagePlaneMetadataExtractor.java:319)
Caused by: java.lang.IllegalArgumentException: -2.77894E9 must not be null or non-negative.
at ome.xml.model.primitives.NonNegativeFloat.(NonNegativeFloat.java:48)
at ome.xml.model.primitives.PositiveFloat.(PositiveFloat.java:46)
at ome.xml.model.primitives.PositiveFloat.valueOf(PositiveFloat.java:64)
at ome.xml.model.Pixels.update(Pixels.java:259)
at ome.xml.model.Pixels.(Pixels.java:166)
at ome.xml.model.Image.update(Image.java:355)
at ome.xml.model.Image.(Image.java:157)
at ome.xml.model.OME.update(OME.java:304)
at ome.xml.model.OME.(OME.java:151)
at ome.xml.meta.OMEXMLMetadataRoot.(OMEXMLMetadataRoot.java:68)
at loci.formats.services.OMEXMLServiceImpl.createRoot(OMEXMLServiceImpl.java:379)
… 3 more

In addition, there’s a “Pipeline error” window showing up with the following report:

Traceback (most recent call last):
File “/home/xxx/CellProfiler/cellprofiler/gui/cornerbuttonmixin.py”, line 113, in on_corner_left_mouse_up
self.fn_clicked()
File “/home/xxx/CellProfiler/cellprofiler/gui/moduleview.py”, line 3931, in on_corner_button_clicked
fn_clicked()
File “/home/xxx/CellProfiler/cellprofiler/modules/metadata.py”, line 916, in update_table
for ipd in self.pipeline.get_image_plane_details(self.workspace):
File “/home/xxx/CellProfiler/cellprofiler/pipeline.py”, line 2725, in get_image_plane_details
self.__prepare_run_module(“Metadata”, workspace)
File “/home/xxx/CellProfiler/cellprofiler/pipeline.py”, line 2691, in __prepare_run_module
return modules[0].prepare_run(workspace)
File “/home/xxx/CellProfiler/cellprofiler/modules/metadata.py”, line 731, in prepare_run
url_array, metadata_array, key_set)
File “/usr/local/lib/python2.7/dist-packages/javabridge/jutil.py”, line 885, in call
result = fn(*nice_args)
File “/usr/local/lib/python2.7/dist-packages/javabridge/jutil.py”, line 852, in fn
raise JavaException(x)
JavaException: java.lang.IllegalArgumentException: -2.77894E9 must not be null or non-negative.


Another stack (T=25, C=2, Z=1) opens in CP as T=49, C=1, Z=1 with the following output in the console:

Exception in thread “Thread-0” java.io.EOFException: Attempting to read beyond end of file.
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
… 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread “Thread-0” java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
at loci.formats.tiff.TiffParser.(TiffParser.java:113)
at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType


#5

The error stacks ending in FlowSightReader actually don’t mean that anything is wrong; it’s just a bug that we’re working on fixing. Sorry for the confusion!

If CP isn’t reading the image metadata correctly in terms of timepoints vs channels, you’ll either need to export to individual tifs or use ImageJ/FIJI to export your full stack with the ‘corrected’ metadata. Depending on how comfortable you are with scripting, this is a script I wrote for another user to change timepoints to channels in her data set; you’d need to make some modifications for your purposes but the bones of the script are there.

Still not sure about the ‘-2.77894E9’ error, since I can’t reproduce it on my own machine and haven’t found evidence of it on the Forum or the GitHub. Sorry!


#6

Ok, thanks for suggestions. I think it might be a problem of the way I prepare these stacks. They are actually stitched together from individual tiffs using bfconvert command line tool. We just prefer to store and analyse stacks with all time points and channels per field of view.


#7

They are actually stitched together from individual tiffs using bfconvert command line tool. We just prefer to store and analyse stacks with all time points and channels per field of view.

Totally get that. Maybe try a different tool? ImageJ/FIJI can definitely import an image sequence and then turn it into a hyperstack with whatever z, t, c, etc dimensions you give it. Sorry!


#8

so this acknowledged as a problem in CP?

i have a similar situation, though nothing is wrong with the metadata. i’ve created a single large OME-TIFF per stage position in order to improve network performance (rather than using thousands of smaller files)

the tif stores a time lapse movie with 4 channels and >2000 time points. it also uses IFD references to previous planes because all channels are not present at every time point - in this example, channel 0 has a real file (IFD 1) whereas channel 1 refers to IFD 0, which is a blank pane:

 <TiffData FirstC="0" FirstT="1" FirstZ="1" IFD="1">
    <UUID FileName="161210_p0403.tiff">
       urn:uuid:6faffa7f-b17c-4566-a8c8-46b044951777
    </UUID>
 </TiffData>
 <TiffData FirstC="1" FirstT="1" FirstZ="1" IFD="0">
    <UUID FileName="161210_p0403.tiff">
       urn:uuid:6faffa7f-b17c-4566-a8c8-46b044951777
    </UUID>
 </TiffData>

it opens fine in FIJI and Omero, and the XML validates using OME command line tools, but in CP, extracting metadata from this single tif only finds channel 0.

not sure why this should work in FIJI and not CP unless there is an issue with the version of bioformats itself. i’m using the latest stable versions of each.

do you suggest that i create one OME-TIFF per channel instead, which means i’ll use a companion OME-XML file?

also, complicating the effort to track down version info is the following bug: when i select Help->About…, i get an exception:

Traceback (most recent call last):
  File "cellprofiler\gui\cpframe.pyc", line 996, in __on_help_about
  File "wx\_controls.pyc", line 997, in __init__
  File "encodings\cp1252.pyc", line 15, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 865: character maps to <undefined>

then when i click “Send report…”, i get “Filed to upload, server reported code 405”


#9

i just tried loading this OME example file and have the same problem. here is part of its XML:

<Pixels BigEndian="true" DimensionOrder="XYZCT" ID="Pixels:0" SizeC="3" SizeT="7" SizeX="439" SizeY="167" SizeZ="1" Type="int8">

  • Images: add tif
  • Metadata: extract metadata from image file headers
  • click Update metadata
  • click Update -> only channel 0 appears, and has 21 rows (SizeT * SizeC – i.e. it is interpreting OME’s own sample data file incorrectly by assigning all planes to one channel)
  • NamesAndTypes -> metadata does have C matching 1 (or even 0, the only one appearing in the Metadata module)
  • click Update -> “Sorry your pipeline doesn’t produce any matching images”

i also see this (like the original poster):

    Exception in thread "Thread-0" loci.common.services.ServiceException: java.lang.
IllegalArgumentException: -1.13623475E12 must not be null or non-negative.
        at loci.formats.services.OMEXMLServiceImpl.createRoot(OMEXMLServiceImpl.java:384)
        at loci.formats.services.OMEXMLServiceImpl.createOMEXMLRoot(OMEXMLServiceImpl.java:350)
        at org.cellprofiler.imageset.ImageFile.setXMLDocument(ImageFile.java:166)
        at org.cellprofiler.imageset.ImagePlaneMetadataExtractor.extract(ImagePlaneMetadataExtractor.java:319)
Caused by: java.lang.IllegalArgumentException: -1.13623475E12 must not be null or non-negative.
        at ome.xml.model.primitives.NonNegativeFloat.<init>(NonNegativeFloat.java:48)
        at ome.xml.model.primitives.PositiveFloat.<init>(PositiveFloat.java:46)
        at ome.xml.model.primitives.PositiveFloat.valueOf(PositiveFloat.java:64)
        at ome.xml.model.Pixels.update(Pixels.java:259)
        at ome.xml.model.Pixels.<init>(Pixels.java:166)
        at ome.xml.model.Image.update(Image.java:355)
        at ome.xml.model.Image.<init>(Image.java:157)
        at ome.xml.model.OME.update(OME.java:304)
        at ome.xml.model.OME.<init>(OME.java:151)
        at ome.xml.meta.OMEXMLMetadataRoot.<init>(OMEXMLMetadataRoot.java:68)
        at loci.formats.services.OMEXMLServiceImpl.createRoot(OMEXMLServiceImpl.java:379)