Olympus IX81 slow, unresponsive and unstable

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Olympus IX81 slow, unresponsive and unstable

Peterbauer Thomas
Hi all,

we are trying to run MM-2.0 on a motorized Olympus IX81 within a
moderately complex system (real time controller board, MT20 illumination
system with filterwheel, attenuator, 2nd shutter, Orca-ER camera, XY
stage; Windows 7 64-bit, MM 2.0 builds up to last week). It works in
principle, but the system is slow, the GUI often unresponsive and we
have frequent MDA crashes. Listening into the communication between the
components, we noticed a hell of traffic at occasions where one would
not  expect it, with frequent response time outs,  GUI
rebuilds/refreshes and EDT hangs (despite increasing the recommended
timeout of the IX81 hub 5-fold). In live mode, for example, the
objective turret and z focus limits (!) are more or less continuously
interrogated, and changing a simple property (such as a demo state
device which we use as dummy for an unmotorized auxiliary magnification)
causes an avelanche of queries and responses between MM and the
microscope. Any ideas/hacks (howsoever nasty) to reduce this burden?
Rebuilding a hacked MM from source would not be a big deal.

Best,
Thomas
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Olympus IX81 slow, unresponsive and unstable

Chris Weisiger
Hi Thomas,

On Wed, May 18, 2016 at 10:59 AM, Peterbauer Thomas <[hidden email]> wrote:
Hi all,

we are trying to run MM-2.0 on a motorized Olympus IX81 within a
moderately complex system (real time controller board, MT20 illumination
system with filterwheel, attenuator, 2nd shutter, Orca-ER camera, XY
stage; Windows 7 64-bit, MM 2.0 builds up to last week). It works in
principle, but the system is slow, the GUI often unresponsive and we
have frequent MDA crashes. Listening into the communication between the
components, we noticed a hell of traffic at occasions where one would
not  expect it, with frequent response time outs,  GUI
rebuilds/refreshes and EDT hangs (despite increasing the recommended
timeout of the IX81 hub 5-fold). In live mode, for example, the
objective turret and z focus limits (!) are more or less continuously
interrogated, and changing a simple property (such as a demo state
device which we use as dummy for an unmotorized auxiliary magnification)
causes an avelanche of queries and responses between MM and the
microscope. Any ideas/hacks (howsoever nasty) to reduce this burden?
Rebuilding a hacked MM from source would not be a big deal.

This appears to have been a bug in the Snap/Live display: as it generates metadata for each image in the display, it was querying the Core for the current X/Y/Z stage position, which in turn was querying the hardware.

I've submitted a fix that will cause µManager to instead use cached values for the stage position (as well as the image bit depth, which was also being queried). This should vastly decrease the amount of traffic you're seeing. Please let me know if it doesn't fully resolve your problem! The fix will be in tonight's nightly build (dated 20160518).

Thanks for the bug report, and I apologize for the inconvenience.

-Chris


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Olympus IX81 slow, unresponsive and unstable

Peterbauer Thomas
Hi Chris,

On 2016-05-18 23:29, Chris Weisiger wrote:
Hi Thomas,

On Wed, May 18, 2016 at 10:59 AM, Peterbauer Thomas <<mailto:[hidden email]>[hidden email]<mailto:[hidden email]>> wrote:
Hi all,

we are trying to run MM-2.0 on a motorized Olympus IX81 within a
moderately complex system (real time controller board, MT20 illumination
system with filterwheel, attenuator, 2nd shutter, Orca-ER camera, XY
stage; Windows 7 64-bit, MM 2.0 builds up to last week). It works in
principle, but the system is slow, the GUI often unresponsive and we
have frequent MDA crashes. Listening into the communication between the
components, we noticed a hell of traffic at occasions where one would
not  expect it, with frequent response time outs,  GUI
rebuilds/refreshes and EDT hangs (despite increasing the recommended
timeout of the IX81 hub 5-fold). In live mode, for example, the
objective turret and z focus limits (!) are more or less continuously
interrogated, and changing a simple property (such as a demo state
device which we use as dummy for an unmotorized auxiliary magnification)
causes an avelanche of queries and responses between MM and the
microscope. Any ideas/hacks (howsoever nasty) to reduce this burden?
Rebuilding a hacked MM from source would not be a big deal.

This appears to have been a bug in the Snap/Live display: as it generates metadata for each image in the display, it was querying the Core for the current X/Y/Z stage position, which in turn was querying the hardware.

I've submitted a fix that will cause µManager to instead use cached values for the stage position (as well as the image bit depth, which was also being queried). This should vastly decrease the amount of traffic you're seeing. Please let me know if it doesn't fully resolve your problem! The fix will be in tonight's nightly build (dated 20160518).

Thanks for the bug report, and I apologize for the inconvenience.

-Chris

Thx for the quick response, but unfortunately it does not solve the problem. Heavy traffic goes on and on and on...
I add an excerpt from the CoreLog from a short live mode session over a few seconds (the whole log file is huge), where 1OB is objective #1, 1SHUT is the transmitted light shutter and so on. When running exposure times of 5 or 10 ms, the objective turret, for example, is queried about every second, unless there's a FetchResponse-Timeout(). Changing  a single property when idle (not in live mode) of whatever device results in a similar avalanche of queries of microscope properties (including objective and reflector turret, condenser, halogen lamp...).

2016-05-19T09:51:51.976599 tid4544 [IFO,dev:Objective] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:51.976599 tid4544 [IFO,dev:Objective] ExecuteQuery-Sending: "1OB?"
2016-05-19T09:51:52.006601 tid4544 [IFO,dev:Objective] FetchResponse-Received: "1OB 1"
2016-05-19T09:51:52.076605 tid4544 [IFO,dev:Objective] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:52.650638 tid4544 [IFO,dev:Reflector] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:52.650638 tid4544 [IFO,dev:Reflector] ExecuteQuery-Sending: "1MU?"
2016-05-19T09:51:52.690640 tid4544 [IFO,dev:Reflector] FetchResponse-Received: "1MU 2"
2016-05-19T09:51:52.750644 tid4544 [IFO,dev:Reflector] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:52.819648 tid4544 [IFO,dev:TL Shutter] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:52.819648 tid4544 [IFO,dev:TL Shutter] ExecuteQuery-Sending: "1SHUT1?"
2016-05-19T09:51:52.859650 tid4544 [IFO,dev:TL Shutter] FetchResponse-Received: "1SHUT1 OUT"
2016-05-19T09:51:52.929654 tid4544 [IFO,dev:TL Shutter] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:53.058661 tid4544 [IFO,dev:Objective] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:53.058661 tid4544 [IFO,dev:Objective] ExecuteQuery-Sending: "1OB?"
2016-05-19T09:51:53.088663 tid4544 [IFO,dev:Objective] FetchResponse-Received: "1OB 1"
2016-05-19T09:51:53.158667 tid4544 [IFO,dev:Objective] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:53.229671 tid468 [IFO,dev:TL Shutter] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:53.429683 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout(1SHUT1)
2016-05-19T09:51:53.430683 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.441683 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.452684 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.463684 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.474685 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.485686 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.496686 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.507687 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.518688 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.529688 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.540689 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.551690 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.562690 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.573691 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.584691 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.595692 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.606693 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.617693 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.628694 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.639695 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.650695 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.661696 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.672696 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.682697 tid468 [IFO,dev:TL Shutter] FetchResponse-Received: "1SHUT1 +"
2016-05-19T09:51:53.683697 tid468 [IFO,dev:TL Shutter] FetchResponse-Timeout()
2016-05-19T09:51:53.753701 tid468 [IFO,dev:TL Shutter] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:53.972714 tid4304 [IFO,dev:TL Shutter] FetchResponse-Received: "1LOG +"
2016-05-19T09:51:53.972714 tid4304 [IFO,dev:TL Shutter] ExecuteQuery-Sending: "1SHUT1?"
2016-05-19T09:51:54.012716 tid4304 [IFO,dev:TL Shutter] FetchResponse-Received: "1SHUT1 IN"
2016-05-19T09:51:54.072719 tid4304 [IFO,dev:TL Shutter] FetchResponse-Received: "1LOG +"


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Olympus IX81 slow, unresponsive and unstable

Mark Tsuchida-3
Hi Thomas,

On Thu, May 19, 2016 at 1:47 AM, Peterbauer Thomas
<[hidden email]> wrote:
> On 2016-05-18 23:29, Chris Weisiger wrote:
> On Wed, May 18, 2016 at 10:59 AM, Peterbauer Thomas <<mailto:[hidden email]>[hidden email]<mailto:[hidden email]>> wrote:
> we are trying to run MM-2.0 on a motorized Olympus IX81 within a
> moderately complex system (real time controller board, MT20 illumination
> system with filterwheel, attenuator, 2nd shutter, Orca-ER camera, XY
> stage; Windows 7 64-bit, MM 2.0 builds up to last week). It works in
> principle, but the system is slow
[...]
> Thx for the quick response, but unfortunately it does not solve the problem. Heavy traffic goes on and on and on...
> I add an excerpt from the CoreLog from a short live mode session over a few seconds (the whole log file is huge), where 1OB is objective #1, 1SHUT is the transmitted light shutter and so on. When running exposure times of 5 or 10 ms, the objective turret, for example, is queried about every second, unless there's a FetchResponse-Timeout(). Changing  a single property when idle (not in live mode) of whatever device results in a similar avalanche of queries of microscope properties (including objective and reflector turret, condenser, halogen lamp...).
[...]

There may multiple different issues here. Some of it might depend on
the details of how you set up your configuration groups. But the first
thing to try is to set the IX81's "Control" property to "Computer"
(not "Manual+Computer").

Best,
Mark

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Olympus IX81 slow, unresponsive and unstable

Peterbauer Thomas
Hi Mark,

On 2016-05-19 18:31, Mark Tsuchida wrote:

> Hi Thomas,
>
> On Thu, May 19, 2016 at 1:47 AM, Peterbauer Thomas
> <[hidden email]> wrote:
>> On 2016-05-18 23:29, Chris Weisiger wrote:
>> On Wed, May 18, 2016 at 10:59 AM, Peterbauer Thomas <<mailto:[hidden email]>[hidden email]<mailto:[hidden email]>> wrote:
>> we are trying to run MM-2.0 on a motorized Olympus IX81 within a
>> moderately complex system (real time controller board, MT20 illumination
>> system with filterwheel, attenuator, 2nd shutter, Orca-ER camera, XY
>> stage; Windows 7 64-bit, MM 2.0 builds up to last week). It works in
>> principle, but the system is slow
> [...]
>> Thx for the quick response, but unfortunately it does not solve the problem. Heavy traffic goes on and on and on...
>> I add an excerpt from the CoreLog from a short live mode session over a few seconds (the whole log file is huge), where 1OB is objective #1, 1SHUT is the transmitted light shutter and so on. When running exposure times of 5 or 10 ms, the objective turret, for example, is queried about every second, unless there's a FetchResponse-Timeout(). Changing  a single property when idle (not in live mode) of whatever device results in a similar avalanche of queries of microscope properties (including objective and reflector turret, condenser, halogen lamp...).
> [...]
>
> There may multiple different issues here. Some of it might depend on
> the details of how you set up your configuration groups. But the first
> thing to try is to set the IX81's "Control" property to "Computer"
> (not "Manual+Computer").

IX81 in "Computer" mode only (no manual control) does not help (even if
it would, it would not be a viable option, because in live mode you
usually want to have at least manual control over the focus drive). I
strongly suspect that it's an MM-2.0 issue rather than my configurations
(which are admittedly complex), because there's much less traffic
between the core and the microscope hub/device adapters when I setup a
similar configuration with MM-1.4. I have not yet all tests I want to
run, but in live mode (particularily bad with MM-2.0), there is complete
silence in MM-1.4, as one would expect: the camera acquires one image
after the other, as it should, and the microscope status is never
queried by the core (unless one manually changes something); there is
zero reason for the core to suspect that something changed if there is
no user interaction and if the microscope does not report any property
changes.

Best,
Thomas

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Olympus IX81 slow, unresponsive and unstable

Chris Weisiger
Hi Thomas,

On Thu, May 19, 2016 at 1:46 PM, Peterbauer Thomas <[hidden email]> wrote:

I strongly suspect that it's an MM-2.0 issue rather than my configurations
(which are admittedly complex), because there's much less traffic
between the core and the microscope hub/device adapters when I setup a
similar configuration with MM-1.4.

Could you please clarify precisely which versions you are testing with? 2.0 builds should all include a date in the name (e.g. "2.0.0-beta3-20160518"), and for 1.4 there should be a "patch" version number, like 1.4.23. Both 1.4 and 2.0 encompass lengthy development periods, so it makes a very big difference if you are talking about, say, 1.4.18 vs. 1.4.23, or 2.0 20150901 vs. 2.0 20160518.

Thanks!

-Chris


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Olympus IX81 slow, unresponsive and unstable

Peterbauer Thomas
Hi Chris,

On 2016-05-19 22:52, Chris Weisiger wrote:
On Thu, May 19, 2016 at 1:46 PM, Peterbauer Thomas <[hidden email]<mailto:[hidden email]>> wrote:

I strongly suspect that it's an MM-2.0 issue rather than my configurations
(which are admittedly complex), because there's much less traffic
between the core and the microscope hub/device adapters when I setup a
similar configuration with MM-1.4.

Could you please clarify precisely which versions you are testing with?

The versions were 2.0.0-beta3-20160518 and 1.4.23-20160410. Today, I installed the latest 2.0 nightly build (2.0.0-beta3-20160519) and I think I could nail down the problem a bit more. The minimum setup to observe exhaustive traffic in live mode includes just the Olympus IX81 hub, the camera (Hamamatsu Orca-ER, in my case), and the filter cube (reflector turret), the basic component for a channel group (config file at the end of this mail). Live mode works still fine (no heavy traffic) if one does *not* define a channel group. If one does, the live mode yields and endless sequence of queries and responses in roughly 100-ms intervals (corresponding to the exposure time):

2016-05-20T13:32:05.698442 tid4364 [IFO,dev:FilterCube] ExecuteQuery-Sending: "1MU?"
2016-05-20T13:32:05.738444 tid4364 [IFO,dev:FilterCube] FetchResponse-Received: "1MU 2"
2016-05-20T13:32:05.880452 tid4364 [IFO,dev:FilterCube] ExecuteQuery-Sending: "1MU?"
2016-05-20T13:32:05.930455 tid4364 [IFO,dev:FilterCube] FetchResponse-Received: "1MU 2"
2016-05-20T13:32:06.072463 tid4364 [IFO,dev:FilterCube] ExecuteQuery-Sending: "1MU?"
2016-05-20T13:32:06.112466 tid4364 [IFO,dev:FilterCube] FetchResponse-Received: "1MU 2"
2016-05-20T13:32:06.255474 tid4364 [IFO,dev:FilterCube] ExecuteQuery-Sending: "1MU?"
2016-05-20T13:32:06.305477 tid4364 [IFO,dev:FilterCube] FetchResponse-Received: "1MU 2"
2016-05-20T13:32:06.447485 tid4364 [IFO,dev:FilterCube] ExecuteQuery-Sending: "1MU?"
2016-05-20T13:32:06.487487 tid4364 [IFO,dev:FilterCube] FetchResponse-Received: "1MU 2"

It can be reversed on the fly: upon removing the channel group (choosing the blank field in the changroup combo), everything is fine again and the core remains silent.

As far as I can tell from a few runs, MDA works under these conditions without problems, so there may be more issues arising with more complex setups, but I think this is very strong evidence for at least one severe bug in the channel group handling.

Best,
Thomas



Minimal configuration:
# Generated by Configurator on Fri May 20 12:50:47 CEST 2016

# Reset
Property,Core,Initialize,0

# Devices
Device,COM1,SerialManager,COM1
Device,Orca-ER,HamamatsuHam,HamamatsuHam_DCAM
Device,IX81,Olympus,OlympusHub
Device,FilterCube,Olympus,FilterCube

# Pre-init settings for devices
Property,IX81,Port,COM1

# Pre-init settings for COM ports
Property,COM1,AnswerTimeout,500.0000
Property,COM1,BaudRate,19200
Property,COM1,DelayBetweenCharsMs,0.0000
Property,COM1,Handshaking,Off
Property,COM1,Parity,Even
Property,COM1,StopBits,2
Property,COM1,Verbose,1

# Hub (parent) references

# Initialize
Property,Core,Initialize,1

# Delays

# Focus directions

# Roles
Property,Core,Camera,Orca-ER
Property,Core,AutoShutter,1

# Camera-synchronized devices

# Labels
# FilterCube
Label,FilterCube,5,Position-6
Label,FilterCube,4,Position-5
Label,FilterCube,3,Position-4
Label,FilterCube,2,Position-3
Label,FilterCube,1,Position-2
Label,FilterCube,0,Position-1

# Configuration presets
# Group: Channel
# Preset: Position-5
ConfigGroup,Channel,Position-5,FilterCube,Label,Position-5

# Preset: Position-4
ConfigGroup,Channel,Position-4,FilterCube,Label,Position-4

# Preset: Position-3
ConfigGroup,Channel,Position-3,FilterCube,Label,Position-3

# Preset: Position-2
ConfigGroup,Channel,Position-2,FilterCube,Label,Position-2

# Preset: Position-1
ConfigGroup,Channel,Position-1,FilterCube,Label,Position-1

# Preset: Position-6
ConfigGroup,Channel,Position-6,FilterCube,Label,Position-6


# Group: System
# Preset: Startup



# PixelSize settings

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general