Making a batch process over subfolders



we have a bunch of images organized by subfolders. We would like to run our current pipeline over all of them keeping the subfolders structure. Hence, we would like to have as many output tables as subfolders (one for each subfolder). I had a look at “CreatBatchFiles” module without success. What is the best way of doing it?




Hi Joaquin,
It sounds like you are already aware of the setting in LoadImages that allows you to process images that are in subfolders. It sounds like your main concern is for the data to be placed in separate output tables when exported to a database.

We had some discussion of this issue here in CellProfiler headquarters, and we agreed that it is not a trivial question to have each subfolder produce its own database. However, it is fairly simple to process all the images/subfolders together and export all the data to a single database (using the CellProfiler code as it currently stands), and then use some fairly straightforward SQL commands to extract data from each subfolder into its own table. Or, depending on how you are accessing your data, why not just use WHERE clauses to select data from particular subfolders? That is, why is it so valuable to have distinct tables?

Another option is for you to attempt to adjust the ExportToDatabase module to create individual tables for each subfolder. It seems challenging but potentially doable. It seems like dealing with the issue at the database end of things is much quicker to get your final result.

I hope this gets you started on the right track! Let us know what you decide to do, and if you add a feature we’d be happy to incorporate it into CellProfiler.