Feature request: More control over output file format in MM2.0g

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Feature request: More control over output file format in MM2.0g

Daniel Dickinson
I'm hoping the developers will consider adding more functionality to the Datastore.SaveMode class to give users more control over how images are saved in 2.0gamma.  In particular, I'd like to have the option to save as multi-plane tiff and without creating a new folder for each acquisition. 

For context, I'm trying to update a custom acquisition plugin from 1.4 to 2.0.  My original (v1.4) code was:

// Loop over several pre-defined stage positions
    // Initialize Acquisition
            gui_.openAcquisition(acqName, dir, n, 1, 1, 1, true, false);
            int width = (int) mmc_.getImageWidth();
            int height = (int) mmc_.getImageHeight();
            int bytesPerPixel = (int) mmc_.getBytesPerPixel();
            int bitDepth = (int) mmc_.getImageBitDepth();
            gui_.initializeAcquisition(acqName, width, height, bytesPerPixel, bitDepth);

    // Acquire some time series data here 

    // Save 
            ij.IJ.saveAs("Tiff",dir+"/"+acqName+".tif");
            gui_.getAcquisition(acqName).promptToSave(false);
            gui_.closeAcquisitionWindow(acqName);
            gui_.closeAllAcquisitions();
// end of Loop

This produced a series of tiff images named exactly the way I wanted, and all in a single folder, which was very easy to deal with in my downstream processing (in MATLAB).  


My new code is: 

// Loop over several pre-defined stage positions
    // Initialize Acquisition 
            Datastore store = gui_.data().createRAMDatastore();
            gui_.displays().createDisplay(store);

    // Acquire some time series data here 

    // Save 
            String savePath = dir+"/"+acqName+".tif";
            store.save(Datastore.SaveMode.MULTIPAGE_TIFF, savePath);
            gui_.displays().closeDisplaysFor(store);
            store.close();
// end of Loop

This works, but produces a bunch of folders (one for each stage position) instead of all images in a single folder, and every image has the obnoxious "MMStack_default" appended to the end of the file name.  I just want all of my images together in a single folder, named the way I expect.  Ideally in plain-vanilla TIFF format, although I can deal with OME-TIFF if I have to.

I tried just calling ij.IJ.saveAs, as I did before, but in 2.0gamma this saves a blank image. 

I also thought about using the "to ImageJ" function, accessible from the gear menu of the display window, to transfer the data to ImageJ and then save with ij.IJ.saveAs. But this might be slow or waste memory, and I also couldn't figure out how to do it (whatever code that gear menu is calling doesn't seem to be part of the API).  

Thanks for any updates or suggestions. 

Dan Dickinson
UT Austin


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Loumarx
I second this request, from another perspective.
uManager also breathes new life into old manual scopes with great optics,
but saving file series is still a pain. You either check "Save Images" in
MDA which will give multiple folders per "position" with cumbersome file
names, or you have to send "to ImageJ" to have more control over naming and
file/folder structure.
Thank you



--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Nico Stuurman-2
In reply to this post by Daniel Dickinson
Hi Daniel,

> I'm hoping the developers will consider adding more functionality to
> the Datastore.SaveMode class to give users more control over how
> images are saved in 2.0gamma.  In particular, I'd like to have the
> option to save as multi-plane tiff and without creating a new folder
> for each acquisition.

I am also not completely happy with decisions that were made in the pas,
but developing and especially maintaining a file format for performant
saving of data is lots of meticulous work, and given the current level
of funding for MM, I do not see that happen.  It would be more
interesting to work on readers of the MM file formats for Python (and
Matlab, if someone who really wants to stick to that platform wants to
work on it).  I can also see that a completely new format suitable for
fast storage of large image data sets (perhaps the nd5 format) would be
of interest.
> I tried just calling ij.IJ.saveAs, as I did before, but in 2.0gamma
> this saves a blank image.

You do not have to put the images you acquire in a data store (or if you
do, put them in a RAM Datastore and get them out again as an Image
object with the name "img".  To move an image to the ImageJ side of
things use:

ImageProcessor ip = mm.data().getImageJConverter().createProcessor(img);

then continue saving as you normilly would.

> I also thought about using the "to ImageJ" function, accessible from
> the gear menu of the display window, to transfer the data to ImageJ
> and then save with ij.IJ.saveAs. But this might be slow or waste
> memory, and I also couldn't figure out how to do it (whatever code
> that gear menu is calling doesn't seem to be part of the API).

Not slow, as there is no pixel data copying involved, also as you see
above, it actually is in the API.  You will need to manage metadata
yourself.

Best,



Nico




_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Nico Stuurman-2
In reply to this post by Loumarx
On 1/28/20 6:02 AM, Loumarx wrote:
> I second this request, from another perspective.
> uManager also breathes new life into old manual scopes with great optics,
> but saving file series is still a pain. You either check "Save Images" in
> MDA which will give multiple folders per "position" with cumbersome file
> names, or you have to send "to ImageJ" to have more control over naming and
> file/folder structure.

Did you uncheck the option "Save XY position in separate Image Stack
Files" under Tools > Options?

Probably still not great for your purposes, but it is really hard to
ensure that MM can still read all the variations of file formats that
have been used in the past, so creating more is certainly not the solution.


Best,


Nico

_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Daniel Dickinson
In reply to this post by Nico Stuurman-2
Hi Nico,

Thanks for your reply.  Can you please clarify a couple of points?

> developing and especially maintaining a file format for performant
> saving of data is lots of meticulous work, and given the current level
> of funding for MM, I do not see that happen.

Point well taken.  Obviously you can't go around implementing a bunch of random formats for whoever wants them, and it sounds like there are other ways to get to what I'm after.


> You do not have to put the images you acquire in a data store

Can you please elaborate on this?  The documentation here says "org.micromanager.data.Datastore is a new class that holds the image data and summary metadata of an acquisition. Every acquisition has an associated Datastore, and you can create your own Datastores to perform custom acquisitions."  This sure makes it sound like in 2.0gamma, all acquisitions are supposed to be done using Datastores.   Scriptinterface.openAcquisition and related functions from 1.4 are gone in 2.0, so I don't know how else to capture an image and put it in RAM.  It would be great to know about other options. 


> ...or if you do, put them in a RAM Datastore and get them out again as an Image object with the name "img".

This is exactly the key thing that I can't figure out how to do.  Once acquisition is done and I have a datastore, how do I get the images back out as an image object?  I looked at Datastore.getImages(Coords), but it looks like that just returns one plane at a time so I'd have to write a loop over all images.  Is there a way to dump an entire datastore to an object that ImageJ knows how to deal with?


> To move an image to the ImageJ side of things use:

> ImageProcessor ip = mm.data().getImageJConverter().createProcessor(img);

Perfect, this will do exactly what I want.  It's just the intermediate step of getting the images into an image object that I'm stuck on.

Thanks for your help!
Dan


From: Stuurman, Nico <[hidden email]>
Sent: Tuesday, January 28, 2020 12:28 PM
To: Micro-Manager General <[hidden email]>
Subject: Re: [micro-manager-general] Feature request: More control over output file format in MM2.0g
 
Hi Daniel,

> I'm hoping the developers will consider adding more functionality to
> the Datastore.SaveMode class to give users more control over how
> images are saved in 2.0gamma.  In particular, I'd like to have the
> option to save as multi-plane tiff and without creating a new folder
> for each acquisition.

I am also not completely happy with decisions that were made in the pas,
but developing and especially maintaining a file format for performant
saving of data is lots of meticulous work, and given the current level
of funding for MM, I do not see that happen.  It would be more
interesting to work on readers of the MM file formats for Python (and
Matlab, if someone who really wants to stick to that platform wants to
work on it).  I can also see that a completely new format suitable for
fast storage of large image data sets (perhaps the nd5 format) would be
of interest.
> I tried just calling ij.IJ.saveAs, as I did before, but in 2.0gamma
> this saves a blank image.

You do not have to put the images you acquire in a data store (or if you
do, put them in a RAM Datastore and get them out again as an Image
object with the name "img".  To move an image to the ImageJ side of
things use:

ImageProcessor ip = mm.data().getImageJConverter().createProcessor(img);

then continue saving as you normilly would.

> I also thought about using the "to ImageJ" function, accessible from
> the gear menu of the display window, to transfer the data to ImageJ
> and then save with ij.IJ.saveAs. But this might be slow or waste
> memory, and I also couldn't figure out how to do it (whatever code
> that gear menu is calling doesn't seem to be part of the API).

Not slow, as there is no pixel data copying involved, also as you see
above, it actually is in the API.  You will need to manage metadata
yourself.

Best,



Nico




_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Nico Stuurman-2
Hi Daniel,

> Thanks for your reply.  Can you please clarify a couple of points?

Happy to help point you in useful directions.  It took me quite some
time to wrap my head around the new API in 2.0 (but now most often like
it quite a bit).

> > You do not have to put the images you acquire in a data store
>
> Can you please elaborate on this?  The documentation here
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__micro-2Dmanager.org_wiki_Version-5F2.0-5FAPI-23The-5FDatastore-5Fclass&d=DwMFAw&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=XA97-5e68Og5Z8yAWb0tNA0NDcOYF1IlBHbBHVWMDIU&s=l2PaG6JU7YXxUBfKTKbg0AQ12KZmbO9ZUsD9ZH01UF8&e=>
> says "org.micromanager.data.Datastore
> <http://valelab.ucsf.edu/~MM/doc-2.0.0-alpha/mmstudio/org/micromanager/data/Datastore.html>
> is a new class that holds the image data and summary metadata of an
> acquisition. Every acquisition has an associated Datastore, and you
> can create your own Datastores to perform custom acquisitions."  This
> sure makes it sound like in 2.0gamma, all acquisitions are supposed to
> be done using Datastores.   Scriptinterface.openAcquisition and
> related functions from 1.4 are gone in 2.0, so I don't know how else
> to capture an image and put it in RAM.  It would be great to know
> about other options.

If you do not use the acquisition engine at all, and are snapping
images, something like:

imgs = mm.live().snap(false);  // it is a bit annoying that this returns
a list, but in very rare cases multiple images are returned, so that
case needs to be handles
img = imgs.get(0); // in almost all situations the first image is the
one you want (or actually, that is the only one there is)

>
> > ...or if you do, put them in a RAM Datastore and get them out again as an Image
> object with the name "img".
>
> This is exactly the key thing that I can't figure out how to do.  Once
> acquisition is done and I have a datastore, how do I get the images
> back out as an image object?  I looked at Datastore.getImages(Coords),
> but it looks like that just returns one plane at a time so I'd have to
> write a loop over all images.  Is there a way to dump an entire
> datastore to an object that ImageJ knows how to deal with?

Once you have a Datastore, or even just a DataProvider (Datastore
extends DataProvider, the store lets you "write", the provider lets you
read images), you can use the DataProvider.getImage(Cooords coord)
function to get the image you desire.  So, the art is to construct a
Coord that specifies which image you want.  You get the coords as follows:

Coords.Builder cb = Coordinates.builder();
cb.c(0).t(2).p(0).z(3); // set the Coords.Builder to channel 0,
timepoint 2, position 0, zSlice 3
c = cb.build();
hasImage = dp.hasIMage(c);
if (hasIMage) {
     img = dp.getImage(c);
}

>
> > To move an image to the ImageJ side of things use:
>
> > ImageProcessor ip = mm.data().getImageJConverter().createProcessor(img);
>
> Perfect, this will do exactly what I want.  It's just the intermediate
> step of getting the images into an image object that I'm stuck on.


Hope this gets you going!


Best,


Nico



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Daniel Dickinson
Hi Nico,

This all makes sense.  I also found your code at https://github.com/nicost/micro-manager/blob/ViewerPlusCV/mmstudio/src/main/java/org/micromanager/display/internal/gearmenu/CopyToImageJItem.java that uses an approach like this.  If I interpreted correctly, this is what the gear button "To ImageJ..." item is doing. 

I have what I need now, but to facilitate future work by others, I think it would be helpful to move this function out of the "internal" methods and make it accessible as a Datastore method, so that one could call something like

store.toImageJ();

and thereby dump the entire datastore into an ImageJ window with just one line of code instead of having to write a loop.

Thanks again,
Dan


From: Stuurman, Nico <[hidden email]>
Sent: Tuesday, January 28, 2020 4:55 PM
To: Micro-Manager General <[hidden email]>
Subject: Re: [micro-manager-general] Feature request: More control over output file format in MM2.0g
 
Hi Daniel,

> Thanks for your reply.  Can you please clarify a couple of points?

Happy to help point you in useful directions.  It took me quite some
time to wrap my head around the new API in 2.0 (but now most often like
it quite a bit).

> > You do not have to put the images you acquire in a data store
>
> Can you please elaborate on this?  The documentation here
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__micro-2Dmanager.org_wiki_Version-5F2.0-5FAPI-23The-5FDatastore-5Fclass&d=DwMFAw&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=XA97-5e68Og5Z8yAWb0tNA0NDcOYF1IlBHbBHVWMDIU&s=l2PaG6JU7YXxUBfKTKbg0AQ12KZmbO9ZUsD9ZH01UF8&e=>
> says "org.micromanager.data.Datastore
> <http://valelab.ucsf.edu/~MM/doc-2.0.0-alpha/mmstudio/org/micromanager/data/Datastore.html>
> is a new class that holds the image data and summary metadata of an
> acquisition. Every acquisition has an associated Datastore, and you
> can create your own Datastores to perform custom acquisitions."  This
> sure makes it sound like in 2.0gamma, all acquisitions are supposed to
> be done using Datastores.   Scriptinterface.openAcquisition and
> related functions from 1.4 are gone in 2.0, so I don't know how else
> to capture an image and put it in RAM.  It would be great to know
> about other options.

If you do not use the acquisition engine at all, and are snapping
images, something like:

imgs = mm.live().snap(false);  // it is a bit annoying that this returns
a list, but in very rare cases multiple images are returned, so that
case needs to be handles
img = imgs.get(0); // in almost all situations the first image is the
one you want (or actually, that is the only one there is)

>
> > ...or if you do, put them in a RAM Datastore and get them out again as an Image
> object with the name "img".
>
> This is exactly the key thing that I can't figure out how to do.  Once
> acquisition is done and I have a datastore, how do I get the images
> back out as an image object?  I looked at Datastore.getImages(Coords),
> but it looks like that just returns one plane at a time so I'd have to
> write a loop over all images.  Is there a way to dump an entire
> datastore to an object that ImageJ knows how to deal with?

Once you have a Datastore, or even just a DataProvider (Datastore
extends DataProvider, the store lets you "write", the provider lets you
read images), you can use the DataProvider.getImage(Cooords coord)
function to get the image you desire.  So, the art is to construct a
Coord that specifies which image you want.  You get the coords as follows:

Coords.Builder cb = Coordinates.builder();
cb.c(0).t(2).p(0).z(3); // set the Coords.Builder to channel 0,
timepoint 2, position 0, zSlice 3
c = cb.build();
hasImage = dp.hasIMage(c);
if (hasIMage) {
     img = dp.getImage(c);
}

>
> > To move an image to the ImageJ side of things use:
>
> > ImageProcessor ip = mm.data().getImageJConverter().createProcessor(img);
>
> Perfect, this will do exactly what I want.  It's just the intermediate
> step of getting the images into an image object that I'm stuck on.


Hope this gets you going!


Best,


Nico



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Loumarx
In reply to this post by Nico Stuurman-2
Nico Stuurman-2 wrote
> Did you uncheck the option "Save XY position in separate Image Stack
> Files" under Tools > Options?

Yes, of course. What I mean is: every new MDA acquisition creates its own
folder, with the tiff and the json file. Most users prefer a single folder
with n files instead of a single folder with multiple n subfolders.

Thank you
- L




--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: More control over output file format in MM2.0g

Nico Stuurman-2
In reply to this post by Loumarx
Since I received one or two more emails about this privately:
> I second this request, from another perspective.
> uManager also breathes new life into old manual scopes with great optics,
> but saving file series is still a pain. You either check "Save Images" in
> MDA which will give multiple folders per "position" with cumbersome file
> names, or you have to send "to ImageJ" to have more control over naming and
> file/folder structure.

I completely agree with everyone that this not fun to deal with.
Regretfully, the file organization is caused by decisions made many,
many years ago (believe me, I protested when they were made;) and
changing them now in a manageable way is very difficult / cost
prohibitive.  Best way forward is to help your users create an easy
workflow for downstream processing.

Best,


Nico




_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general