Micromanager 1.4 MDA Dropping frames

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Micromanager 1.4 MDA Dropping frames

AlexFlorea

Hello Micromanager community,

 

We have been trying to get SRRF acquisition working in Micromanager using an Andor iXon Ultra 897. Previously, our lab has used Metamorph to control our microscope and we are able to achieve proper frame rates (52fps). However, when we switched to control with Micromanager 1.4 pushing the camera to higher frame rates results in a significant number of dropped frames. This problem was resolved when we switched to Micromanager 2.0 (20161115), but happened again when we installed the most recent Micromanager 2.0 build. Below is description of our hardware, the issues we have been encountering and our initial steps to remedy the issues.

 

Hardware

Camera: Andor iXon Ultra 897, 2012 model

Pixel type: 16bit

ReadoutMode: 17000 MHz

Full 512x512 image

Shutters: Open

Andor-TimeOut: 10000

Vertical clock voltage: +4

Vertical speed: 0.3

Binning: 1

Triggering: Software and Internal triggering behave the same way

Frame Transfer: Tried both ON/OFF (see below)

Micro-Manager 1.4.23 on a 32-bit Windows 7 with 16GB RAM. 

 

Minimum readout time = 19.73ms so the following tests were done at 20ms exposure to compare the behavior of the software:

 

Software

Time to acquire 50 frames w/ 20ms exposure

Time to acquire 100 frames w/ 20ms exposure

Metamorph – Using stream acquisition

1.023s

2.07s

Micromanager MDA (Andor frame transfer OFF)

2.18s

4.41s

Micromanager MDA (Andor frame transfer ON)

1.07 – dropped 21 frames

2.15s – dropped 44 frames

 

With Frame Transfer ON, the acquisition speed matches the frame rates from Metamorph very closely, but the acquisition starts dropping frames. For example, instead of acquiring all 50 frames, only 29 frames will be stored before an error message appears. Keeping all other settings the same, the higher the exposure time, the fewer frames end up being dropped for a 50 image burst in the Micromanager MDA:

 

Exposure

Dropped Frames

20ms

21

35ms

6

50ms

0

 

1)    We initially thought this was an issue with the circular buffer. However, running the acquisition with the sequence buffer monitor ON shows it only filling up with a maximum of 1-2 images out of the 500 it can hold. It seems the size of the buffer isn’t limiting the acquisition.

2)    The error that message on micromanager when the acquisition is waiting for frames is: “Error: Acquisition failed. java.lang.Exception: Timed out waiting for image to arrive from camera.”

3)    It seems that the computer isn’t grabbing images from the camera fast enough, thus causing multiple frames to be missed. The camera then stops acquiring while the computer is still waiting for frames

4)    Power saving was disabled in the BIOS and in the power plan options with no change to the number of dropped frames. 

5)    The frame dropping was seen in MM2.0 in the most recent nightly build. This however was not seen in Micromanager 2.0 with the 20161115 nightly build. Using this build, the MDA ran at the same framerate as Metamorph.


Thanks,

Alex




_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Micromanager 1.4 MDA Dropping frames

Stuurman, Nico
Hi Alex,

A week or two ago, I worked on a similar issue with Sam Lord.  We
determined that the dropped frames were caused by code (added by Andor)
to measure the sensor temperature for every frame and add it to the
metadata.  You could check if it is the same problem by comparing the
(1.4) nightly build 20180311 and 20180312.  The first one worked fine
for Sam, whereas the second one had a problem very similar to what you
describe.  We noticed that the problem happened with Andor iXons using
USB for data transfer (like yours?), but not for iXons with Cameralink
interface.

I notified Andor, but did not get a response (who knows, the email may
have gotten lost).  You may want to ask Andor directly, and hopefully
that will accelerate things.

The solution is relatively obvious: make temperature logging during
sequence acquisition optional (or - if my analysis is correct - do not
do it for USB-based cameras).

Once there is a solution in 1.4, there is the additional problem of
getting those updates into 2.0.  No updates to the Core of 2.0-beta have
happened since June.  I am contemplating doing manual updates to
2.0-gamma, but the thought of doing so is a bit overwhelming.

Best,

Nico


> Hello Micromanager community,
>
> We have been trying to get SRRF acquisition working in Micromanager
> using an Andor iXon Ultra 897. Previously, our lab has used Metamorph
> to control our microscope and we are able to achieve proper frame
> rates (52fps). However, when we switched to control with Micromanager
> 1.4 pushing the camera to higher frame rates results in a significant
> number of dropped frames. This problem was resolved when we switched
> to Micromanager 2.0 (20161115), but happened again when we installed
> the most recent Micromanager 2.0 build. Below is description of our
> hardware, the issues we have been encountering and our initial steps
> to remedy the issues.
>
> _Hardware_
>
> Camera: Andor iXon Ultra 897, 2012 model
>
> Pixel type: 16bit
>
> ReadoutMode: 17000 MHz
>
> Full 512x512 image
>
> Shutters: Open
>
> Andor-TimeOut: 10000
>
> Vertical clock voltage: +4
>
> Vertical speed: 0.3
>
> Binning: 1
>
> Triggering: Software and Internal triggering behave the same way
>
> Frame Transfer: Tried both ON/OFF (see below)
>
> Micro-Manager 1.4.23 on a 32-bit Windows 7 with 16GB RAM.
>
> Minimum readout time = 19.73ms so the following tests were done at
> 20ms exposure to compare the behavior of the software:
>
> Software
>
>
>
> Time to acquire 50 frames w/ 20ms exposure
>
>
>
> Time to acquire 100 frames w/ 20ms exposure
>
> Metamorph – Using stream acquisition
>
>
>
> 1.023s
>
>
>
> 2.07s
>
> Micromanager MDA (Andor frame transfer OFF)
>
>
>
> 2.18s
>
>
>
> 4.41s
>
> Micromanager MDA (Andor frame transfer ON)
>
>
>
> 1.07 – dropped 21 frames
>
>
>
> 2.15s – dropped 44 frames
>
> With Frame Transfer ON, the acquisition speed matches the frame rates
> from Metamorph very closely, but the acquisition starts dropping
> frames. For example, instead of acquiring all 50 frames, only 29
> frames will be stored before an error message appears. Keeping all
> other settings the same, the higher the exposure time, the fewer
> frames end up being dropped for a 50 image burst in the Micromanager MDA:
>
> Exposure
>
>
>
> Dropped Frames
>
> 20ms
>
>
>
> 21
>
> 35ms
>
>
>
> 6
>
> 50ms
>
>
>
> 0
>
> 1)We initially thought this was an issue with the circular buffer.
> However, running the acquisition with the sequence buffer monitor ON
> shows it only filling up with a maximum of 1-2 images out of the 500
> it can hold. It seems the size of the buffer isn’t limiting the
> acquisition.
>
> 2)The error that message on micromanager when the acquisition is
> waiting for frames is: “Error: Acquisition failed.
> java.lang.Exception: Timed out waiting for image to arrive from camera.”
>
> 3)It seems that the computer isn’t grabbing images from the camera
> fast enough, thus causing multiple frames to be missed. The camera
> then stops acquiring while the computer is still waiting for frames
>
> 4)Power saving was disabled in the BIOS and in the power plan options
> with no change to the number of dropped frames.
>
> 5)The frame dropping was seen in MM2.0 in the most recent nightly
> build. This however was not seen in Micromanager 2.0 with the 20161115
> nightly build. Using this build, the MDA ran at the same framerate as
> Metamorph.
>
>
> Thanks,
>
> Alex
>
>
>
>
>
>
> _______________________________________________
> micro-manager-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/micro-manager-general



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Micromanager 1.4 MDA Dropping frames

AlexFlorea
This post was updated on .
Hi Nico,

Thanks for your response, it was really helpful. I tried the two releases you suggested, and the acquisition worked fine in 20180311, while dropping frames in 20180312. It seems to be running into the same issue.

We currently have the camera connected with USB and Cameralink. I will have to go back and check whether it is actually using the Cameralink for data transfer or it's just transferring over USB.


Best,
Alex

_______________________________________________
micro-manager-general mailing list
micro-manager-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Micromanager 1.4 MDA Dropping frames

Stuurman, Nico
Hi Alex and Sam,

Haven't heard from Andor, but I just added another property to the Andor
adapter code.  By default, the adapter will no longer try to update the
temperature during a sequence acquisition (so, the expectation is that
this will fix the dropped frames problem). For those who want to
register the temperature, there is a property ("Off" by default) that
can be switched to "On" to update temperature readings during sequence
acquisition.  Temperature readings seem to be a problem in any case, as
our second iXon always reports a strange temperature.

The change should be in tonight's build of 1.4.  I'll port it soon to
2.0-gamma.

Best,

Nico




On 11/19/18 3:33 PM, Alexandru Florea wrote:

>
> Hi Nico,
>
> Thanks for your response, it was really helpful. I tried the two
> releases you suggested, and the acquisition worked fine in 20180311,
> while dropping frames in 20180312. It seems to be running into the
> same issue.
>
> We currently have the camera connected with USB and Cameralink. I will
> have to go back and check whether it is actually using the Cameralink
> for data transfer or it's just transferring over USB.
>
>
> Best,
> Alex
>
> ------------------------------------------------------------------------
> *From:* Stuurman, Nico <[hidden email]>
> *Sent:* Friday, November 16, 2018 5:53:44 PM
> *To:* Micro-Manager General
> *Subject:* Re: [micro-manager-general] Micromanager 1.4 MDA Dropping
> frames
> Hi Alex,
>
> A week or two ago, I worked on a similar issue with Sam Lord.  We
> determined that the dropped frames were caused by code (added by Andor)
> to measure the sensor temperature for every frame and add it to the
> metadata.  You could check if it is the same problem by comparing the
> (1.4) nightly build 20180311 and 20180312.  The first one worked fine
> for Sam, whereas the second one had a problem very similar to what you
> describe.  We noticed that the problem happened with Andor iXons using
> USB for data transfer (like yours?), but not for iXons with Cameralink
> interface.
>
> I notified Andor, but did not get a response (who knows, the email may
> have gotten lost).  You may want to ask Andor directly, and hopefully
> that will accelerate things.
>
> The solution is relatively obvious: make temperature logging during
> sequence acquisition optional (or - if my analysis is correct - do not
> do it for USB-based cameras).
>
> Once there is a solution in 1.4, there is the additional problem of
> getting those updates into 2.0.  No updates to the Core of 2.0-beta have
> happened since June.  I am contemplating doing manual updates to
> 2.0-gamma, but the thought of doing so is a bit overwhelming.
>
> Best,
>
> Nico
>
>
> > Hello Micromanager community,
> >
> > We have been trying to get SRRF acquisition working in Micromanager
> > using an Andor iXon Ultra 897. Previously, our lab has used Metamorph
> > to control our microscope and we are able to achieve proper frame
> > rates (52fps). However, when we switched to control with Micromanager
> > 1.4 pushing the camera to higher frame rates results in a significant
> > number of dropped frames. This problem was resolved when we switched
> > to Micromanager 2.0 (20161115), but happened again when we installed
> > the most recent Micromanager 2.0 build. Below is description of our
> > hardware, the issues we have been encountering and our initial steps
> > to remedy the issues.
> >
> > _Hardware_
> >
> > Camera: Andor iXon Ultra 897, 2012 model
> >
> > Pixel type: 16bit
> >
> > ReadoutMode: 17000 MHz
> >
> > Full 512x512 image
> >
> > Shutters: Open
> >
> > Andor-TimeOut: 10000
> >
> > Vertical clock voltage: +4
> >
> > Vertical speed: 0.3
> >
> > Binning: 1
> >
> > Triggering: Software and Internal triggering behave the same way
> >
> > Frame Transfer: Tried both ON/OFF (see below)
> >
> > Micro-Manager 1.4.23 on a 32-bit Windows 7 with 16GB RAM.
> >
> > Minimum readout time = 19.73ms so the following tests were done at
> > 20ms exposure to compare the behavior of the software:
> >
> > Software
> >
> >
> >
> > Time to acquire 50 frames w/ 20ms exposure
> >
> >
> >
> > Time to acquire 100 frames w/ 20ms exposure
> >
> > Metamorph – Using stream acquisition
> >
> >
> >
> > 1.023s
> >
> >
> >
> > 2.07s
> >
> > Micromanager MDA (Andor frame transfer OFF)
> >
> >
> >
> > 2.18s
> >
> >
> >
> > 4.41s
> >
> > Micromanager MDA (Andor frame transfer ON)
> >
> >
> >
> > 1.07 – dropped 21 frames
> >
> >
> >
> > 2.15s – dropped 44 frames
> >
> > With Frame Transfer ON, the acquisition speed matches the frame rates
> > from Metamorph very closely, but the acquisition starts dropping
> > frames. For example, instead of acquiring all 50 frames, only 29
> > frames will be stored before an error message appears. Keeping all
> > other settings the same, the higher the exposure time, the fewer
> > frames end up being dropped for a 50 image burst in the Micromanager
> MDA:
> >
> > Exposure
> >
> >
> >
> > Dropped Frames
> >
> > 20ms
> >
> >
> >
> > 21
> >
> > 35ms
> >
> >
> >
> > 6
> >
> > 50ms
> >
> >
> >
> > 0
> >
> > 1)We initially thought this was an issue with the circular buffer.
> > However, running the acquisition with the sequence buffer monitor ON
> > shows it only filling up with a maximum of 1-2 images out of the 500
> > it can hold. It seems the size of the buffer isn’t limiting the
> > acquisition.
> >
> > 2)The error that message on micromanager when the acquisition is
> > waiting for frames is: “Error: Acquisition failed.
> > java.lang.Exception: Timed out waiting for image to arrive from camera.”
> >
> > 3)It seems that the computer isn’t grabbing images from the camera
> > fast enough, thus causing multiple frames to be missed. The camera
> > then stops acquiring while the computer is still waiting for frames
> >
> > 4)Power saving was disabled in the BIOS and in the power plan options
> > with no change to the number of dropped frames.
> >
> > 5)The frame dropping was seen in MM2.0 in the most recent nightly
> > build. This however was not seen in Micromanager 2.0 with the 20161115
> > nightly build. Using this build, the MDA ran at the same framerate as
> > Metamorph.
> >
> >
> > Thanks,
> >
> > Alex
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > micro-manager-general mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>
>
>
> _______________________________________________
> micro-manager-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>
>
>
>
> _______________________________________________
> micro-manager-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/micro-manager-general



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Micromanager 1.4 MDA Dropping frames

AlexFlorea
This post was updated on .
Hi Nico,


Thanks for the fix. I just tried the new nightly build and everything is working as expected!


Alex
_______________________________________________
micro-manager-general mailing list
micro-manager-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Micromanager 1.4 MDA Dropping frames

Stuurman, Nico

On 11/25/2018 6:29 PM, Alexandru Florea wrote:
Thanks for the fix. I just tried the new nightly build and everything is working as expected!

Great to hear!  Happy all works as expected again.

Best,

Nico




_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general