snapImage() & python freeze

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

snapImage() & python freeze

StevenF
Hello Micromanager community,

I have some issues when using the SnapImage function. It seems, that it's
making my python instance lag/freeze/stall even if it's called in a thread.

here is the code i use to test this bug :

*import MMCorePy
import threading

mmc = MMCorePy.CMMCore()
mmc.loadSystemConfiguration("MYCONFIG.cfg")
mmc.setExposure(10000)
threading.Thread(target=mmc.snapImage).start()
print('yeah')*

This is a bit of an issue because it makes my custom GUI & other things
lag/stall...

I'm using python 3, and the nightly build from
https://valelab4.ucsf.edu/~MM/nightlyBuilds/2.0.0-beta/Windows/ 
MMSetup_64bit_2.0.0-beta3_20171106.exe 06-Nov-2017 22:49   68M

I would use a more up to date version of the beta but from what I can tell
the hasn't been any news on bringing the compiled 3.0 version.
I don't really know where this issue is coming from so any help or quick
testing on your side would be appreciated.

Thanks in advance!

Steven





--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: snapImage() & python freeze

Stuurman, Nico
Hi Steven,

> I have some issues when using the SnapImage function. It seems, that it's
> making my python instance lag/freeze/stall even if it's called in a thread.
>
> here is the code i use to test this bug :
>
> *import MMCorePy
> import threading
>
> mmc = MMCorePy.CMMCore()
> mmc.loadSystemConfiguration("MYCONFIG.cfg")
> mmc.setExposure(10000)
> threading.Thread(target=mmc.snapImage).start()
> print('yeah')*
>
> This is a bit of an issue because it makes my custom GUI & other things
> lag/stall...


mmc.snapImage() is blocking by design.  This mechanism lets you
synchronize exposure with other things.  So, the problem seems to be in
the Threading mechanism that you are using.  I have little or no
experience in Python, but I would first try to replace mmc.snapImage()
with a simple sleep and see if your thread is doing what you would like
it to do.  When I look at the Thread object documentation
(https://docs.python.org/3/library/threading.html#thread-objects), it
does not look like your invocation will do what you would like it to do.

Best,


Nico



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: snapImage() & python freeze

StevenF
Stuurman, Nico wrote
> mmc.snapImage() is blocking by design.

I understand that, but like with the Micromanager GUI, while the acquisition
loop is running i would like to be able to manipulate other parts of my
code. I have made some tests, and if you run a simple time counter  in
parallel to my custom Acquisition loop, well whenever SnapImage() is called
the time counter will lag behind.

In fact, after further investigation, it seems that most if not all the
MMCorePy functions are blocking even if they are called from a thread, which
is from what i gather is one of the correct ways to do parallelism is python
(multiprocessing being the other).
Trying the use Multiprocessing will also not work (from what i gather
compatibility issues between pickle and SWIG)


Stuurman, Nico wrote
> replace mmc.snapImage() with a simple sleep and see if your thread is
> doing what you would like it to do.

It does do what i want it to do.

I don't really know why either threading or multiprocessing isn't working
with MMCorePy and i will try to investigate why on my end. If anyone has any
suggestions i'm all ears!

Thanks for the help Nico!

Stevene



--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: snapImage() & python freeze

Stuurman, Nico
On 1/16/19 3:52 AM, StevenF wrote:
> Stuurman, Nico wrote
>> mmc.snapImage() is blocking by design.
> I don't really know why either threading or multiprocessing isn't working
> with MMCorePy and i will try to investigate why on my end. If anyone has any
> suggestions i'm all ears!

Very interested to hear what you find out!  I only have heard rumors
about Python and multi-threading being a bit of an issue.  It could be
that something needs to be done in the Python Swig bridge to enable
multi-threading (but kind of would have expected others to already have
experienced this issue).  It would be great if this will be taken into
account if/when a Python3 bridge will be build.

Best,

Nico



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: snapImage() & python freeze

Kyle Douglass-2

I have only have heard rumors about Python and multi-threading being a bit of an issue.

The CPython interpeter has an infamous global interpeter lock (GIL) that prevents multiple threads from running at the same time *unless they are I/O bound and explicitly release the GIL in the C layer during I/O.* All Python standard library functions that do I/O release the GIL so that, if you have several I/O bound tasks, you can gain an increase in speed by multithreading.

I have never seen anything remotely related to the GIL in the SWIG interface definition, so I doubt you can gain any benefit from multithreading. I do know of someone using multiprocessing for parallelism in Python-based microscopy acquisition; I can share his name with you if you email me offline. (By the way each interpreter process has its own GIL, so launching multiple interpeter processes is one way to gain parallelism in Python; this is how multiprocessing works.)

Finally, you might be able to improve single threaded concurrency by using the asyncio library and "awaiting" on snapImage, but I am not familiar enough with it to say whether it could work.

I hope this helps clear up some things.

Cheers,
Kyle




_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: snapImage() & python freeze

StevenF
After some more digging it seems there are few issues with MMCorePy and
python when it comes to parallelism.
So first of as i discovered and as Kyle stated, there is GIL in CPython that
essentially prevents the use of mutlithreading. I know that some libraries
have some things in place to bypass/remove the GIL (numpy does this if i
recall), but i haven't done to much digging in this direction.
The easiest solution would be to use use multiprocessing instead of
multithreading. Unfortunately python multiprocessing relies on being able to
'pickle' the objects that are called during the process, and SWIG or at
least how the MMCorePy library has been written isn't pickle-able, rendering
it impossible for me to use multiprocessing.
Making the the current MMCorePy pickle-able is a bit out of my
reach/capabilities.
Last suggestion i had was the use of concurrencies using libraries like
asyncio built from my initial testing it isn't going to work for my case.

i will continue to try and solve this issue as it's a pretty problematic one
for sure :)!

Thanks for the help Kyle and Nico,
Cheers,

Steven



--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general