A few questions about Micro-Manager 2.0gamma

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

A few questions about Micro-Manager 2.0gamma

Kyle Douglass-2
Hi everyone,
I've got just a few questions about MM 2.0gamma.

1. Does the mm2 branch on the primary GitHub repository (https://github.com/micro-manager/micro-manager) correspond to 2.0beta or 2.0gamma?

2. Is there a reason  why the link to the MM 2.0 gamma nightly builds has not been placed on the Micro-Manager wiki? Specifically, I'm referring to https://micro-manager.org/wiki/Version_2.0.

3. I seem to remember a forum post in which someone said that plugins that are written for 2.0beta will not work in 2.0gamma, but I cannot find this post. Can someone confirm the incompatability of 2.0beta and 2.0gamma plugins?

4. If there is an incompatibility, where might I find the 2.0gamma plugin API?

Thanks!
Kyle



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

Seamus Holden
Thanks for raising this Kyle.
I also have a few questions to add:

5) What differences are there between 2.0gamma and 2.0beta? 
6) Why are there two different 2.0 branches? Are they going to merge?
7) Which one - gamma or beta - is currently recommended for use?

On Wed, 16 May 2018 at 11:58 Kyle Douglass <[hidden email]> wrote:
Hi everyone,
I've got just a few questions about MM 2.0gamma.

1. Does the mm2 branch on the primary GitHub repository (https://github.com/micro-manager/micro-manager) correspond to 2.0beta or 2.0gamma?

2. Is there a reason  why the link to the MM 2.0 gamma nightly builds has not been placed on the Micro-Manager wiki? Specifically, I'm referring to https://micro-manager.org/wiki/Version_2.0.

3. I seem to remember a forum post in which someone said that plugins that are written for 2.0beta will not work in 2.0gamma, but I cannot find this post. Can someone confirm the incompatability of 2.0beta and 2.0gamma plugins?

4. If there is an incompatibility, where might I find the 2.0gamma plugin API?

Thanks!
Kyle


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

Nico Stuurman-2
Hi Kyle and Seamus,

All great questions!  I am partially responsible for this, but really
very much want everything back together on one nice, clean branch.  Mark
hopefully will chime in as well, but here are my answers.

First, a bit of background:  While re-evaluating the beta branch about a
year ago, Mark realized that there were quite a few api calls that were
far from optimal.  Since the code was still beta, he started
re-architecting in his mm2-next branch.  Since there were practical
issues with beta, and since I knew the mark/mm2-next was the way
forward, I decided late last year to pick up mm2-next, fix everything
that did not work (and there was a lot of that because of the
re-architecting), move to Java 8 and include the ClearVolume viewer
plugin, and start building this branch (ViewerPlusCV,
https://github.com/nicost/micro-manager/tree/ViewerPlusCV) as
2.0-gamma.  A couple of months ago, Mark merged a lot of the
ViewerPlusCV branch back into mm2-next, and with a bit of work we
hopefully will soon be able to start using the mm2-next branch as the
final 2.0 beta version and all merge into that branch.

In the mean time, our lab is running 2.0gamma quite successfully
(although the horrible bug of occasionally stalling live mode is still
there).


On 5/16/18 4:04 AM, Seamus Holden wrote:
>
>     1. Does the mm2 branch on the primary GitHub repository
>     (https://github.com/micro-manager/micro-manager) correspond to
>     2.0beta or 2.0gamma?
>

2.0beta.  The source for 2.0gamma lives here:
https://github.com/nicost/micro-manager/tree/ViewerPlusCV

>     2. Is there a reason  why the link to the MM 2.0 gamma nightly
>     builds has not been placed on the Micro-Manager wiki?
>     Specifically, I'm referring to
>     https://micro-manager.org/wiki/Version_2.0.
>

2.0gamma is my own "rogue" build.  It is basically a build for our lab
that I make available to anyone who would like to use it.  A nice entry
point to find the builds (plus my tutorial for the Singapore Focus On
Microscopy Meeting) is here: http://valelab4.ucsf.edu/~nstuurman/fom/
It is up to Mark to decide where the "official" release should point.

>     3. I seem to remember a forum post in which someone said that
>     plugins that are written for 2.0beta will not work in 2.0gamma,
>     but I cannot find this post. Can someone confirm the
>     incompatability of 2.0beta and 2.0gamma plugins?
>

There are changes in the api between 2.0beta and 2.0gamma, so it is
likely that plugins will break.  These changes should be easily fixable,
and I am happy to help find the easiest fix.

>     4. If there is an incompatibility, where might I find the 2.0gamma
>     plugin API?
>

Easiest is to work from source:
https://github.com/nicost/micro-manager/tree/ViewerPlusCV.  I can try to
figure out how to generate Javadoc, but that will take me a bit of time.

> 5) What differences are there between 2.0gamma and 2.0beta?

In most cases, Mark deprecated functions/classes and replaced them with
newly named ones.  At one point, I got so confused by the long list of
deprecated functions that I removed a bunch.  I think that Mark kept
those in his mm2-next, so come to think about it, plugins may compile
fine with mm2-next (which however, can not yet be used for any practical
work).  Compiling them with ViewerPlusCV will either work, or there will
be some missing functions/classes, in which case you will need to find
the corresponding new versions.

> 6) Why are there two different 2.0 branches? Are they going to merge?
I hope I explained that all!  I very much hope this will all merge as
soon as possible, I really want to get out of this as soon as possible,
but it is out of my hands now.

> 7) Which one - gamma or beta - is currently recommended for use?
We run everything with gamma.  I will only work on gamma myself (recent
new features include live ratio viewer, photon transfer curve generator,
updates to the Localization Microscopy plugin for distance
measuerements, on top of Java 8 and the ClearVolume-based viewer). Can't
wait for it to merge with Mark's mm2-next though.

Hope this explains things!

Best,


NIco



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

Mark Tsuchida-3
Hi Kyle and Seamus,

Sorry for causing this confusing situation. It's all as Nico explained.

Plugins written for mm2 will not work after mm2-next is merged back into mm2. However, fixing such plugins will not require any structural changes in most cases. There will be straightforward instructions on how to update plugins.

When? With the caveat that this schedule has been delayed multiple times and may well be delayed further, I'm currently aiming for end of June for the merge of mm2-next into mm2.

There are a few things that need to be finished before we can merge everything back into mm2, but the major ones are these:

1) File i/o (including metadata handling) needs an overhaul, since we now need to read at least 3 different formats (1.4, early 2.0 beta, late 2.0 beta) and make sure we write a future-proof format (which I don't expect to be much different from the 2.0 beta formats). Files written by gamma might also be different, adding to the work. I'm getting a little tired of dealing with the (frankly unwarranted) proliferation of these formats, so I feel very strongly about getting this right before widespread use.

2) The API needs a comprehensive review, so that APIs in mm2 are not broken without good reason, and also some new APIs may be moved back to experimental status.

3) A few minor things need to be forward-ported from mm2

I would currently recommend gamma only for advanced users who can troubleshoot things at code level. At least, I'm not able to provide support for it.

Best,
Mark

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

JonD
Administrator
Maybe this deserves its own thread but Mark mentioned an important topic:


Mark Tsuchida-3 wrote
> 1) File i/o (including metadata handling) needs an overhaul, since we now
> need to read at least 3 different formats (1.4, early 2.0 beta, late 2.0
> beta) and make sure we write a future-proof format (which I don't expect
> to
> be much different from the 2.0 beta formats). Files written by gamma might
> also be different, adding to the work. I'm getting a little tired of
> dealing with the (frankly unwarranted) proliferation of these formats, so
> I
> feel very strongly about getting this right before widespread use.

Will there be some sort of modular file writer/reader structure in 2.0 that
is easy to extend to add new formats?

I can foresee use cases for writing directly to hierarchical formats (e.g.
BigDataViewer), lossless compressed formats (e.g. Keller Lab Block), or
lossy compressed formats (e.g. "Adaptive Particle Representation").  It
seems that Micro-Manager itself should be agnostic as to the details of the
backing store.  In other words it should be able accommodate many different
formats as long as there was a pre-defined interface that each writer/reader
would implement.  I'm not necessarily asking the core developers to
implement all these different formats, but make it easy for others to
implement them as a sort of plugin.

From what little I know the SCIFIO project aims are similar, maybe
Micro-Manager could piggy-back on that?

Would love to hear thoughts and become better educated.

-------------------------------------------
Jon Daniels
Applied Scientific Instrumentation
29391 West Enid Rd, Eugene, OR 97402
Phone: (541) 461-8181 x118
-------------------------------------------



--
Sent from: http://micro-manager.3463995.n2.nabble.com/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

Mark Tsuchida-3
Hi Jon,

On Thu, May 17, 2018 at 7:53 AM, JonD <[hidden email]> wrote:
Mark Tsuchida-3 wrote
> 1) File i/o (including metadata handling) needs an overhaul, since we now
> need to read at least 3 different formats (1.4, early 2.0 beta, late 2.0
> beta) and make sure we write a future-proof format (which I don't expect
> to
> be much different from the 2.0 beta formats). Files written by gamma might
> also be different, adding to the work. I'm getting a little tired of
> dealing with the (frankly unwarranted) proliferation of these formats, so
> I
> feel very strongly about getting this right before widespread use.

Will there be some sort of modular file writer/reader structure in 2.0 that
is easy to extend to add new formats?

This is definitely a future goal, but may not be easy to do in the initial 2.0 (we don't want to push back 2.0 further for the purpose of optimizing for this type of extensibility). In 2.0beta there is Datastore, which could be extended in principle, but the API design is not optimal at the moment. We probably need a layer closer to the file i/o. (There is the "Storage" interface, but it is a little too strongly tied to the Datastore mechanism and the current file i/o implementations.)
 
I can foresee use cases for writing directly to hierarchical formats (e.g.
BigDataViewer), lossless compressed formats (e.g. Keller Lab Block), or
lossy compressed formats (e.g. "Adaptive Particle Representation").  It
seems that Micro-Manager itself should be agnostic as to the details of the
backing store.  In other words it should be able accommodate many different
formats as long as there was a pre-defined interface that each writer/reader
would implement.  I'm not necessarily asking the core developers to
implement all these different formats, but make it easy for others to
implement them as a sort of plugin.

I agree. However, something has to be done to deal with the fact that these non-native formats aren't capable of backing a Datastore on their own. (For example, you can store µManager metadata in them.) Which means we probably need a slightly different abstraction for non-native datastores -- somewhat analogous to the distinction between "Save/Open" and "Export/Import" in many programs.
 
From what little I know the SCIFIO project aims are similar, maybe
Micro-Manager could piggy-back on that?

Indeed. I had a very good discussion with Curtis Rueden a few weeks ago related to this. For starters, it looks like it will be possible to extend the file i/o code I'm currently working on into something generic enough for Scifio and/or ImageJ to also use for TIFF I/O. It will be nice to see the sharing of such an important component. Once something like that is ready, it will be natural to also support the other Scifio-supported formats. Of course, this will be a topic for the next year or so and will not be part of what I do to get mm2-next ready for merge.

Best,
Mark

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

Kyle Douglass-2
In reply to this post by Mark Tsuchida-3
Hi Mark and Nico,
Thank you very much for the explanations; this clears up quite a bit of confusion on my part. I very much appreciate the work that you have been putting into this!

I asked because I am doing a bit of forward planning for our lab's plugins; at the moment, we have a dependency that requires a JRE >= 7, so MM2.0gamma is something I look forward to ;)

Cheers,
Kyle

On Thu, May 17, 2018 at 12:14 AM, Mark Tsuchida <[hidden email]> wrote:
Hi Kyle and Seamus,

Sorry for causing this confusing situation. It's all as Nico explained.

Plugins written for mm2 will not work after mm2-next is merged back into mm2. However, fixing such plugins will not require any structural changes in most cases. There will be straightforward instructions on how to update plugins.

When? With the caveat that this schedule has been delayed multiple times and may well be delayed further, I'm currently aiming for end of June for the merge of mm2-next into mm2.

There are a few things that need to be finished before we can merge everything back into mm2, but the major ones are these:

1) File i/o (including metadata handling) needs an overhaul, since we now need to read at least 3 different formats (1.4, early 2.0 beta, late 2.0 beta) and make sure we write a future-proof format (which I don't expect to be much different from the 2.0 beta formats). Files written by gamma might also be different, adding to the work. I'm getting a little tired of dealing with the (frankly unwarranted) proliferation of these formats, so I feel very strongly about getting this right before widespread use.

2) The API needs a comprehensive review, so that APIs in mm2 are not broken without good reason, and also some new APIs may be moved back to experimental status.

3) A few minor things need to be forward-ported from mm2

I would currently recommend gamma only for advanced users who can troubleshoot things at code level. At least, I'm not able to provide support for it.

Best,
Mark

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

Nico Stuurman-2
On 5/17/18 12:11 AM, Kyle Douglass wrote:
> Thank you very much for the explanations; this clears up quite a bit
> of confusion on my part. I very much appreciate the work that you have
> been putting into this!
>
> I asked because I am doing a bit of forward planning for our lab's
> plugins; at the moment, we have a dependency that requires a JRE >= 7,
> so MM2.0gamma is something I look forward to ;)

Unlike Mark, I have no qualms about using MM2.0-gamma.  It is as close
as possible to the future 2.0 release, and is tested daily in our lab. 
There are sure to be bugs, and sure to be (hopefully small) future
changes to the api, but it is the best way forward that I can see.

Also, referring to discussions of SciFIO in this thread.  I worked on a
SciFIO-based file opener in 2.0-gamma and discovered that there is no
build-in support for opening multi-dimensional data in SciFIO.  Also,
the project is basically unsupported, as the documentation is bare, and
it takes many months to weeks to get questions answered.  Unless major
changes happen in that project, I do advice everyone to stay away from
it (hey, I invested a couple of weeks trying to make it work, so hope
this advice can save some people time).

Best,

Nico


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: A few questions about Micro-Manager 2.0gamma

Kyle Douglass-2
Hi Nicole,

On Thu, May 17, 2018, 18:22 Nico Stuurman <[hidden email]> wrote:

Unlike Mark, I have no qualms about using MM2.0-gamma.  It is as close
as possible to the future 2.0 release, and is tested daily in our lab. 
There are sure to be bugs, and sure to be (hopefully small) future
changes to the api, but it is the best way forward that I can see.

Actually I tried out MM2g today and it's pretty slick! The live view felt smooth and very responsive. Better yet, I only had to change one call to a Datastore event handler to get our plugins setup for the new API.

Thanks again for all the hard work on this.

Cheers,
Kyle

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general