alternative data saving formats like HDF5

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

alternative data saving formats like HDF5

JonD
Administrator
Hello list,

Has anyone tried adding the ability to save data directly to formats other
than TIFF?  Or maybe somebody can comment on how difficult this would be to
implement, either in 1.4 or 2.0?  What are the key things that need to be
worked out?  Pardon my ignorance on these matters.

I see a trend of acquiring large datasets that will be post-processed in
BigDataViewer and related Fiji plugins that prefer HDF5/XML.  As I
understand it HDF5 is a lossless way of reshuffling the data hierarchically
so it can easily be viewed at the desired resolution and is technically
similar to how Imaris and other such programs store image data.  At the
moment we acquire multi-GB datasets in Micro-Manager in TIFF, export to
HDF5/XML in Fiji, and then proceed with analysis in Fiji.  It seems to be
better if Micro-Manager could just save as HDF5 then we would save time and
disk space.  I suppose since TIFF is such a universal format there hopefully
would be a way to get back to TIFF if you needed to (or else save it in TIFF
as well as HDF5 which only solves half the problem).

I also have heard rumblings of people working on compressing microscopy
images, not unlike how MP3 files are much smaller than raw audio files and
are perfectly adequate for most purposes.  I know in many situations no loss
can be tolerated, but I also see plenty of situations where the tradeoff is
worth it.  I'm guessing working on such compression schemes would love to
add their algorithms to acquisition software directly and there's no better
place to start than Micro-Manager.

Thanks for helping to educate me.  Happy holidays everyone!

Jon

PS: I found a similar post from a year ago but nobody replied:
http://micro-manager.3463995.n2.nabble.com/Streaming-images-into-HDF-or-other-container-file-types-td7587870.html

--------------------------------------------
Jon Daniels
Applied Scientific Instrumentation
29391 West Enid Rd, Eugene, OR 97402
Phone: (541) 461-8181 x118
--------------------------------------------



--
Sent from: http://micro-manager.3463995.n2.nabble.com/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: alternative data saving formats like HDF5

Nico Stuurman-2
Hi Jon,

On 12/22/17 10:32 AM, JonD wrote:
 > Has anyone tried adding the ability to save data directly to formats
other
 > than TIFF?  Or maybe somebody can comment on how difficult this would
be to
 > implement, either in 1.4 or 2.0?  What are the key things that need to be
 > worked out?  Pardon my ignorance on these matters.

My understanding was that in 2.0, one can implement a Datastore
(https://github.com/nicost/micro-manager/blob/ViewerPlusCV/mmstudio/src/main/java/org/micromanager/data/Datastore.java),
<https://github.com/nicost/micro-manager/blob/ViewerPlusCV/mmstudio/src/main/java/org/micromanager/data/Datastore.java>

and plug it in somewhere so that the user can choose it. Looking at the
code, I now see that "Custom Datastore implementations are not supported
(consider creating a custom storage instead)."  It is not clear to me
how "Storage" and "Datastore" are related, they seem to be more or less
the same thing.

In any case, this is where you should be looking, and input in
untangling this and providing better documentation will be highly
appreciated!
 > I see a trend of acquiring large datasets that will be post-processed in
 > BigDataViewer and related Fiji plugins that prefer HDF5/XML.  As I
 > understand it HDF5 is a lossless way of reshuffling the data
hierarchically
 > so it can easily be viewed at the desired resolution and is technically
 > similar to how Imaris and other such programs store image data.

As an added complication, I see HDF5 more as a an alternative file
system (that is portable).  So, storing in HDF5 does not necessarily
solve the problem, it would need to be in HDF5 compatiblw with whatever
reader you want to use.

 >At the moment we acquire multi-GB datasets in Micro-Manager in TIFF,
 > export to
 > HDF5/XML in Fiji, and then proceed with analysis in Fiji.  It seems to be
 > better if Micro-Manager could just save as HDF5 then we would save
time and
 > disk space.  I suppose since TIFF is such a universal format there
hopefully
 > would be a way to get back to TIFF if you needed to (or else save it
in TIFF
 > as well as HDF5 which only solves half the problem).

You may want to look at Henry's Micro-Magellan plugin.  He has a way to
save data such that they are compatible with BigDataviewer (see:
https://micro-manager.org/wiki/MicroMagellan#Using_Micro-Magellan_data_with_BigDataViewer).
<https://micro-manager.org/wiki/MicroMagellan#Using_Micro-Magellan_data_with_BigDataViewer>

I am not sure if it is a HDF5-based format, but at the very least, it
can be directly opened by teh Big DataViewer and has multi-resolution
storage.

 > I also have heard rumblings of people working on compressing microscopy
 > images, not unlike how MP3 files are much smaller than raw audio
files and
 > are perfectly adequate for most purposes.  I know in many situations
no loss
 > can be tolerated, but I also see plenty of situations where the
tradeoff is
 > worth it.  I'm guessing working on such compression schemes would love to
 > add their algorithms to acquisition software directly and there's no
better
 > place to start than Micro-Manager.

I certainly hope that data storage and reading in MM 2.0 is isolated
enough to make it straight forward to add other formats, but have never
tried myself.

Best,

Nico
.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general