Just wanted to share some problems we’ve experienced using Export to Database and some issues with teh way this is logically done.
We have a user with many images in multiple folders from many experiments
We had set up a pipeline which is quite robust and works on his images very well
The output sought is simply an area meaurement of the secondary objects identified
As the user has many images and will run many more, we had hoped that he could simply append the results to the existing SQLite database
We had expected the last point to work as you offer the option to either DROP the tables or “Overwrite as necessary”
Knowing that when we load images we have the following, quoted from another post here
Each folder is unique and the image name is also unique in our case so we assumed that this uniqueness was used in the database as well.
We found out that by doing several runs, Cell Profiler overwrites the results in the Per_Object and Per_Image tables because the only thing taken into account there is the ImageID and Object IDs as unique columns and these always reset for each new run…
Moreover, using anything to identify the current run other than append a prefix to the tables (Which makes extraction all the more tedious) nothing else works.
Experiment name does nothing:
Checking the database after two runs shows that it did not update or append another experiment name to the corresponding database table. Why would this be included if it does not help with multiple experiments in the same database?
What would make sense is to have an option to APPEND the results to the table, or if you are going to proceed to overwriting, use the same uniqueness criteria as you do for the groups when looking for metadata tags. If you append, the ImageID and ObjectIDs should start at the last object and not be overwritten. Or some combination of unique experiment ID and imageId…
Otherwise we have to create a DB file for each run, which kind of kills the purpose of a database.
Please let me know if we are doing something wrong or what you do to append data to an existing database, so that we can correct this error as soon as possible.
Thank you and best regards.